2024-09-15T20:14:30.854 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T20:14:30.871 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T20:14:30.955 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604 branch: main description: rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905604' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAMOovC30gXvL370lpa50lS5EQ/JX7oF8ii7aeE/RBbF+BDUVyfTd1pj6LBOVaiuFnjHeCX8Ayuo9xboStDQrjA= tasks: - install: null - ceph: conf: osd: osd max pg log entries: 5 osd min pg log entries: 5 create_rbd_pool: false log-ignorelist: - Error - overall HEALTH_ - \(OBJECT_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_BACKFILLFULL\) - \(POOL_NEARFULL\) - \(SLOW_OPS\) - \(TOO_FEW_PGS\) - Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - backfill_toofull: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T20:14:30.956 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-15T20:14:30.956 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-15T20:14:30.956 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T20:14:30.958 INFO:teuthology.task.internal:Checking packages... 2024-09-15T20:14:30.979 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-15T20:14:30.979 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T20:14:30.980 INFO:teuthology.packaging:ref: None 2024-09-15T20:14:30.980 INFO:teuthology.packaging:tag: None 2024-09-15T20:14:30.980 INFO:teuthology.packaging:branch: main 2024-09-15T20:14:30.980 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:14:30.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-15T20:14:31.225 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4930.gfc3e0bff 2024-09-15T20:14:31.227 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T20:14:31.232 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T20:14:31.232 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T20:14:31.262 INFO:teuthology.task.internal:Saving configuration 2024-09-15T20:14:31.274 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T20:14:31.288 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T20:14:31.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 20:03:56.678196', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAMOovC30gXvL370lpa50lS5EQ/JX7oF8ii7aeE/RBbF+BDUVyfTd1pj6LBOVaiuFnjHeCX8Ayuo9xboStDQrjA='} 2024-09-15T20:14:31.316 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T20:14:31.321 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-15T20:14:31.322 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T20:14:31.376 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f29d5dd7ac0>, signals=[15]) 2024-09-15T20:14:31.376 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T20:14:31.387 INFO:teuthology.task.internal:Opening connections... 2024-09-15T20:14:31.388 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:14:31.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:14:31.463 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T20:14:31.469 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-09-15T20:14:31.559 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-09-15T20:14:31.560 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-09-15T20:14:31.617 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-09-15T20:14:31.618 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-09-15T20:14:31.619 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T20:14:31.619 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-09-15T20:14:31.619 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T20:14:31.619 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T20:14:31.619 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T20:14:31.620 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-09-15T20:14:31.675 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T20:14:31.683 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T20:14:31.704 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T20:14:31.705 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-09-15T20:14:31.722 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T20:14:31.729 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T20:14:31.729 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-09-15T20:14:31.869 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T20:14:31.901 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T20:14:31.920 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T20:14:31.920 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T20:14:31.920 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T20:14:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:14:31.939 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-15T20:14:31.997 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-503.el9.x86_64 2024-09-15T20:14:31.997 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-503.el9.x86_64 2024-09-15T20:14:31.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-09-15T20:14:35.360 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.2 MB 00:01 2024-09-15T20:14:37.237 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-15T20:14:41.429 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-15T20:14:42.838 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 19 kB 00:00 2024-09-15T20:14:43.617 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-15T20:14:49.173 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T20:14:51.188 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T20:14:51.188 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T20:14:51.289 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:14:51.297 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T20:14:51.298 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:14:51.299 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:14:51.299 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:14:51.299 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-09-15T20:14:51.299 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:14:51.299 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 79 M 2024-09-15T20:14:51.299 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 126 M 2024-09-15T20:14:51.299 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:14:51.483 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 167 kB/s | 23 kB 00:00 2024-09-15T20:14:51.791 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-15T20:14:52.150 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 43 MB/s | 29 MB 00:00 2024-09-15T20:14:52.600 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-15T20:14:52.600 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:14:52.600 INFO:teuthology.orchestra.run.smithi129.stdout:Total 61 MB/s | 79 MB 00:01 2024-09-15T20:14:53.287 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:14:53.643 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:14:53.643 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:14:54.581 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:14:54.581 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:14:57.020 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:14:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T20:14:57.817 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:15:00.633 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:15:00.751 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:15:05.597 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:15:05.720 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:15:10.586 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:15:49.171 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:15:49.210 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:15:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:15:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T20:15:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:15:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:15:50.524 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:15:50.865 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T20:15:50.941 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T20:15:50.942 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T20:15:50.942 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T20:15:50.942 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T20:15:50.942 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T20:15:50.942 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T20:15:50.942 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T20:15:50.942 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T20:15:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T20:15:51.001 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T20:15:51.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-09-15T20:15:51.035 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 82.el9 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8274670 2024-09-15T20:15:51.036 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-09-15T20:15:51.037 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T20:15:51.037 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T20:15:51.037 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T20:15:51.037 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T20:15:51.037 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-09-15T20:15:51.037 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-09-15T20:15:51.037 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-09-15T20:15:51.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:15:51.039 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T20:15:51.040 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T20:15:51.040 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-505.el9.x86_64 2024-09-15T20:15:51.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T20:15:51.669 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-09-15T20:15:52.910 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T20:15:52.945 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-09-15T20:15:52.947 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-15T20:15:52.962 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.IOZn87VaAL 2024-09-15T20:15:52.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IOZn87VaAL 2024-09-15T20:15:53.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.IOZn87VaAL 2024-09-15T20:15:53.166 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.IOZn87VaAL is 6KB 2024-09-15T20:15:53.215 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.IOZn87VaAL 2024-09-15T20:15:53.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T20:15:53.293 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T20:15:53.293 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T20:15:53.293 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T20:15:53.293 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T20:15:53.295 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T20:15:53.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-09-15T20:16:23.485 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T20:16:23.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:16:23.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:16:42.045 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-15T20:16:51.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:16:51.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:16:54.109 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-15T20:17:06.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:17:06.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:17:09.181 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-15T20:17:24.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:17:24.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:17:27.265 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-15T20:17:45.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:17:45.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:17:45.668 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-15T20:17:46.004 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:17:46.005 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T20:17:46.005 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-15T20:17:46.025 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-505.el9.x86_64 2024-09-15T20:17:46.026 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T20:17:46.026 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T20:17:46.026 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T20:17:47.027 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T20:17:47.027 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T20:17:47.096 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-09-15T20:17:47.119 DEBUG:teuthology.parallel:result is None 2024-09-15T20:17:47.119 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T20:17:47.128 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T20:17:47.128 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T20:17:47.151 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T20:17:47.157 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T20:17:47.210 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T20:17:47.210 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T20:17:47.246 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T20:17:47.259 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T20:17:47.259 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T20:17:47.319 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:17:47.330 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:17:47.331 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T20:17:47.340 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T20:17:47.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T20:17:47.402 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T20:17:47.413 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T20:17:47.414 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T20:17:47.459 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:17:47.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:17:47.591 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:17:47.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:17:47.714 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:17:47.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T20:17:47.780 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-09-15T20:17:47.866 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T20:17:48.122 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T20:17:48.129 INFO:teuthology.task.internal:Starting timer... 2024-09-15T20:17:48.129 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T20:17:48.180 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T20:17:48.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-09-15T20:17:48.285 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-09-15T20:17:48.286 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T20:17:48.287 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-09-15T20:17:48.327 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-09-15T20:17:48.328 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-09-15T20:17:48.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T20:17:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:17:48.393 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T20:17:48.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-09-15T20:17:48.463 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T20:17:48.477 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T20:17:48.491 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T20:17:48.492 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T20:22:41.944 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-09-15T20:22:41.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:22:41.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:22:42.078 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-15T20:22:42.160 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-15T20:22:42.160 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T20:22:42.182 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T20:22:42.182 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T20:22:42.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:22:42.233 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T20:22:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T20:22:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-09-15T20:22:42.285 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-09-15T20:22:42.297 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T20:22:42.311 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T20:22:42.368 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-15T20:22:42.376 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T20:22:42.376 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-15T20:22:42.376 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:22:42.376 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:22:42.376 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:22:42.376 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:22:42.376 INFO:teuthology.run_tasks:Running task install... 2024-09-15T20:22:42.390 DEBUG:teuthology.task.install:project ceph 2024-09-15T20:22:42.390 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-15T20:22:42.390 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-15T20:22:42.391 INFO:teuthology.task.install:Using flavor: default 2024-09-15T20:22:42.398 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T20:22:42.398 INFO:teuthology.task.install:extra packages: [] 2024-09-15T20:22:42.408 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'tag': None, 'wait_for_package': False} 2024-09-15T20:22:42.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:22:42.557 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/ 2024-09-15T20:22:42.557 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4930.gfc3e0bff 2024-09-15T20:22:42.670 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T20:22:42.671 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:22:42.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T20:22:42.698 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T20:22:42.698 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fc3e0bff7468fd55478adbae7f75508b749124b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T20:22:42.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T20:22:42.841 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T20:22:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-09-15T20:22:42.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-09-15T20:22:43.150 INFO:teuthology.orchestra.run.smithi129.stdout:45 files removed 2024-09-15T20:22:43.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw 2024-09-15T20:22:43.759 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 286 kB/s | 84 kB 00:00 2024-09-15T20:22:44.043 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-15T20:22:44.318 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-15T20:22:44.788 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-15T20:22:46.728 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-15T20:22:51.045 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T20:22:52.846 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-15T20:22:53.800 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-15T20:22:59.298 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-15T20:23:01.692 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 11 M 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.5 M 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-15T20:23:01.707 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 22 M 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 25 k 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 755 k 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T20:23:01.708 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T20:23:01.709 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T20:23:01.709 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 508 k 2024-09-15T20:23:01.715 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T20:23:01.715 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-15T20:23:01.715 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T20:23:01.715 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 45 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 143 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 320 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 305 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 99 k 2024-09-15T20:23:01.716 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:23:01.717 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:01.718 INFO:teuthology.orchestra.run.smithi129.stdout:Install 31 Packages 2024-09-15T20:23:01.718 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2024-09-15T20:23:01.718 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:01.718 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 62 M 2024-09-15T20:23:01.718 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:23:02.660 INFO:teuthology.orchestra.run.smithi129.stdout:(1/33): ceph-base-19.3.0-4930.gfc3e0bff.el9.x86 7.8 MB/s | 5.5 MB 00:00 2024-09-15T20:23:02.710 INFO:teuthology.orchestra.run.smithi129.stdout:(2/33): ceph-selinux-19.3.0-4930.gfc3e0bff.el9. 506 kB/s | 25 kB 00:00 2024-09-15T20:23:02.802 INFO:teuthology.orchestra.run.smithi129.stdout:(3/33): libcephfs2-19.3.0-4930.gfc3e0bff.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-15T20:23:02.893 INFO:teuthology.orchestra.run.smithi129.stdout:(4/33): libradosstriper1-19.3.0-4930.gfc3e0bff. 5.4 MB/s | 508 kB 00:00 2024-09-15T20:23:03.036 INFO:teuthology.orchestra.run.smithi129.stdout:(5/33): ceph-radosgw-19.3.0-4930.gfc3e0bff.el9. 10 MB/s | 11 MB 00:01 2024-09-15T20:23:03.086 INFO:teuthology.orchestra.run.smithi129.stdout:(6/33): python3-ceph-argparse-19.3.0-4930.gfc3e 905 kB/s | 45 kB 00:00 2024-09-15T20:23:03.145 INFO:teuthology.orchestra.run.smithi129.stdout:(7/33): python3-ceph-common-19.3.0-4930.gfc3e0b 2.4 MB/s | 143 kB 00:00 2024-09-15T20:23:03.203 INFO:teuthology.orchestra.run.smithi129.stdout:(8/33): python3-cephfs-19.3.0-4930.gfc3e0bff.el 2.7 MB/s | 161 kB 00:00 2024-09-15T20:23:03.270 INFO:teuthology.orchestra.run.smithi129.stdout:(9/33): python3-rados-19.3.0-4930.gfc3e0bff.el9 4.7 MB/s | 320 kB 00:00 2024-09-15T20:23:03.379 INFO:teuthology.orchestra.run.smithi129.stdout:(10/33): librgw2-19.3.0-4930.gfc3e0bff.el9.x86_ 11 MB/s | 5.5 MB 00:00 2024-09-15T20:23:03.404 INFO:teuthology.orchestra.run.smithi129.stdout:(11/33): python3-rbd-19.3.0-4930.gfc3e0bff.el9. 2.2 MB/s | 305 kB 00:00 2024-09-15T20:23:03.429 INFO:teuthology.orchestra.run.smithi129.stdout:(12/33): python3-rgw-19.3.0-4930.gfc3e0bff.el9. 1.9 MB/s | 99 kB 00:00 2024-09-15T20:23:03.638 INFO:teuthology.orchestra.run.smithi129.stdout:(13/33): ceph-common-19.3.0-4930.gfc3e0bff.el9. 13 MB/s | 22 MB 00:01 2024-09-15T20:23:04.272 INFO:teuthology.orchestra.run.smithi129.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 123 kB/s | 107 kB 00:00 2024-09-15T20:23:04.372 INFO:teuthology.orchestra.run.smithi129.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 181 kB/s | 171 kB 00:00 2024-09-15T20:23:04.405 INFO:teuthology.orchestra.run.smithi129.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 209 kB/s | 160 kB 00:00 2024-09-15T20:23:04.873 INFO:teuthology.orchestra.run.smithi129.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 75 kB/s | 45 kB 00:00 2024-09-15T20:23:05.289 INFO:teuthology.orchestra.run.smithi129.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 331 kB/s | 292 kB 00:00 2024-09-15T20:23:05.406 INFO:teuthology.orchestra.run.smithi129.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 641 kB/s | 662 kB 00:01 2024-09-15T20:23:05.506 INFO:teuthology.orchestra.run.smithi129.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 67 kB/s | 42 kB 00:00 2024-09-15T20:23:06.090 INFO:teuthology.orchestra.run.smithi129.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 52 kB/s | 36 kB 00:00 2024-09-15T20:23:06.115 INFO:teuthology.orchestra.run.smithi129.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 79 kB/s | 47 kB 00:00 2024-09-15T20:23:06.149 INFO:teuthology.orchestra.run.smithi129.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 257 kB/s | 221 kB 00:00 2024-09-15T20:23:06.200 INFO:teuthology.orchestra.run.smithi129.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-15T20:23:06.225 INFO:teuthology.orchestra.run.smithi129.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-15T20:23:06.250 INFO:teuthology.orchestra.run.smithi129.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 286 kB/s | 26 kB 00:00 2024-09-15T20:23:06.275 INFO:teuthology.orchestra.run.smithi129.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-15T20:23:06.309 INFO:teuthology.orchestra.run.smithi129.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-15T20:23:06.368 INFO:teuthology.orchestra.run.smithi129.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-15T20:23:06.435 INFO:teuthology.orchestra.run.smithi129.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-15T20:23:06.486 INFO:teuthology.orchestra.run.smithi129.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-15T20:23:06.694 INFO:teuthology.orchestra.run.smithi129.stdout:(32/33): librados2-19.3.0-4930.gfc3e0bff.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-15T20:23:06.752 INFO:teuthology.orchestra.run.smithi129.stdout:(33/33): librbd1-19.3.0-4930.gfc3e0bff.el9.x86_ 10 MB/s | 3.2 MB 00:00 2024-09-15T20:23:06.758 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:23:06.758 INFO:teuthology.orchestra.run.smithi129.stdout:Total 12 MB/s | 62 MB 00:05 2024-09-15T20:23:06.875 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:23:06.912 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:23:06.912 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:23:07.570 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:23:07.570 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:23:09.095 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:23:09.178 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-15T20:23:09.216 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-15T20:23:09.241 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-15T20:23:09.264 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-15T20:23:09.284 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-15T20:23:09.474 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-15T20:23:09.528 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-15T20:23:09.904 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-15T20:23:09.976 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T20:23:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T20:23:10.143 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 8/35 2024-09-15T20:23:10.169 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-15T20:23:10.234 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-15T20:23:10.270 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-15T20:23:10.325 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 11/35 2024-09-15T20:23:10.400 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-15T20:23:10.422 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-15T20:23:10.486 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-15T20:23:10.529 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-15T20:23:10.646 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-15T20:23:10.689 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-15T20:23:10.749 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-15T20:23:10.778 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-15T20:23:10.808 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-15T20:23:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-15T20:23:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-15T20:23:11.207 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-15T20:23:11.254 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-15T20:23:11.346 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-15T20:23:11.388 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 24/35 2024-09-15T20:23:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-15T20:23:11.934 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-15T20:23:11.965 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-15T20:23:12.020 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-15T20:23:12.094 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 28/35 2024-09-15T20:23:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 29/35 2024-09-15T20:23:14.471 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T20:23:14.515 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T20:23:15.108 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T20:23:15.152 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-15T20:23:15.227 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-15T20:23:15.227 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T20:23:15.227 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T20:23:15.227 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:15.262 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T20:23:25.431 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T20:23:25.432 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-09-15T20:23:25.432 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-09-15T20:23:25.432 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-09-15T20:23:25.432 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-09-15T20:23:25.432 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-09-15T20:23:25.432 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-09-15T20:23:25.433 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-09-15T20:23:25.433 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:26.444 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-15T20:23:26.470 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-15T20:23:26.470 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:23:26.470 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T20:23:26.470 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:23:26.470 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:23:26.471 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:26.471 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-15T20:23:26.556 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-15T20:23:26.556 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 6/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T20:23:27.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 8/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 9/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 10/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 11/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 13/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-15T20:23:27.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-15T20:23:27.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-15T20:23:27.750 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T20:23:27.750 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-15T20:23:27.750 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/35 2024-09-15T20:23:28.745 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T20:23:28.745 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:28.745 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.746 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T20:23:28.747 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:28.748 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:23:29.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test 2024-09-15T20:23:29.696 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:30 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:23:29.829 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 46 M 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T20:23:29.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:29.831 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:23:29.831 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:29.831 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-09-15T20:23:29.831 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:29.831 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 46 M 2024-09-15T20:23:29.831 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 201 M 2024-09-15T20:23:29.831 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:23:30.329 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): libcephsqlite-19.3.0-4930.gfc3e0bff.el9. 746 kB/s | 161 kB 00:00 2024-09-15T20:23:30.678 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 182 kB/s | 64 kB 00:00 2024-09-15T20:23:30.728 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 493 kB/s | 304 kB 00:00 2024-09-15T20:23:32.137 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): ceph-test-19.3.0-4930.gfc3e0bff.el9.x86_ 22 MB/s | 46 MB 00:02 2024-09-15T20:23:32.139 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:23:32.139 INFO:teuthology.orchestra.run.smithi129.stdout:Total 20 MB/s | 46 MB 00:02 2024-09-15T20:23:32.167 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:23:32.187 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:23:32.188 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:23:32.557 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:23:32.557 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:23:33.073 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:23:33.162 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T20:23:33.203 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T20:23:33.230 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-15T20:23:37.602 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-15T20:23:37.680 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-15T20:23:40.107 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-15T20:23:40.107 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/4 2024-09-15T20:23:40.108 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/4 2024-09-15T20:23:40.108 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T20:23:40.413 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T20:23:40.413 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:40.413 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:23:40.413 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:40.414 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:40.414 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T20:23:40.414 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T20:23:40.414 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:40.414 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:23:40.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph 2024-09-15T20:23:41.293 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:42 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:23:41.439 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:23:41.441 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:41.441 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:23:41.441 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:41.441 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 6.6 k 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 2.4 M 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 1.5 M 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 291 k 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 4.8 M 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 17 M 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T20:23:41.442 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T20:23:41.443 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T20:23:41.444 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T20:23:41.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T20:23:41.445 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T20:23:41.445 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:41.445 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:23:41.445 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:41.445 INFO:teuthology.orchestra.run.smithi129.stdout:Install 37 Packages 2024-09-15T20:23:41.449 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:41.451 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 31 M 2024-09-15T20:23:41.451 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 109 M 2024-09-15T20:23:41.451 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:23:42.040 INFO:teuthology.orchestra.run.smithi129.stdout:(1/37): ceph-19.3.0-4930.gfc3e0bff.el9.x86_64.r 45 kB/s | 6.6 kB 00:00 2024-09-15T20:23:42.207 INFO:teuthology.orchestra.run.smithi129.stdout:(2/37): ceph-mgr-19.3.0-4930.gfc3e0bff.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-15T20:23:42.257 INFO:teuthology.orchestra.run.smithi129.stdout:(3/37): ceph-mds-19.3.0-4930.gfc3e0bff.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-15T20:23:42.366 INFO:teuthology.orchestra.run.smithi129.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4930.gfc3e 2.6 MB/s | 291 kB 00:00 2024-09-15T20:23:42.516 INFO:teuthology.orchestra.run.smithi129.stdout:(5/37): ceph-mon-19.3.0-4930.gfc3e0bff.el9.x86_ 10 MB/s | 4.8 MB 00:00 2024-09-15T20:23:42.541 INFO:teuthology.orchestra.run.smithi129.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-15T20:23:42.587 INFO:teuthology.orchestra.run.smithi129.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.3 MB/s | 106 kB 00:00 2024-09-15T20:23:42.629 INFO:teuthology.orchestra.run.smithi129.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-15T20:23:42.804 INFO:teuthology.orchestra.run.smithi129.stdout:(9/37): ceph-osd-19.3.0-4930.gfc3e0bff.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-15T20:23:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 623 kB/s | 125 kB 00:00 2024-09-15T20:23:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 3.3 MB/s | 1.2 MB 00:00 2024-09-15T20:23:42.972 INFO:teuthology.orchestra.run.smithi129.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-15T20:23:43.539 INFO:teuthology.orchestra.run.smithi129.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 61 kB/s | 35 kB 00:00 2024-09-15T20:23:43.564 INFO:teuthology.orchestra.run.smithi129.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 257 kB/s | 188 kB 00:00 2024-09-15T20:23:43.614 INFO:teuthology.orchestra.run.smithi129.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 237 kB/s | 172 kB 00:00 2024-09-15T20:23:44.164 INFO:teuthology.orchestra.run.smithi129.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 85 kB/s | 51 kB 00:00 2024-09-15T20:23:44.190 INFO:teuthology.orchestra.run.smithi129.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 119 kB/s | 77 kB 00:00 2024-09-15T20:23:44.232 INFO:teuthology.orchestra.run.smithi129.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-09-15T20:23:44.266 INFO:teuthology.orchestra.run.smithi129.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-15T20:23:44.299 INFO:teuthology.orchestra.run.smithi129.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 670 kB/s | 43 kB 00:00 2024-09-15T20:23:44.325 INFO:teuthology.orchestra.run.smithi129.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-15T20:23:44.350 INFO:teuthology.orchestra.run.smithi129.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 121 kB/s | 22 kB 00:00 2024-09-15T20:23:44.366 INFO:teuthology.orchestra.run.smithi129.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-09-15T20:23:44.392 INFO:teuthology.orchestra.run.smithi129.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.3 MB/s | 23 kB 00:00 2024-09-15T20:23:44.409 INFO:teuthology.orchestra.run.smithi129.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.3 MB/s | 19 kB 00:00 2024-09-15T20:23:44.426 INFO:teuthology.orchestra.run.smithi129.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 233 kB/s | 18 kB 00:00 2024-09-15T20:23:44.443 INFO:teuthology.orchestra.run.smithi129.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-09-15T20:23:44.460 INFO:teuthology.orchestra.run.smithi129.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-15T20:23:44.493 INFO:teuthology.orchestra.run.smithi129.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-09-15T20:23:44.519 INFO:teuthology.orchestra.run.smithi129.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-15T20:23:44.536 INFO:teuthology.orchestra.run.smithi129.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-15T20:23:44.561 INFO:teuthology.orchestra.run.smithi129.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-15T20:23:44.578 INFO:teuthology.orchestra.run.smithi129.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-09-15T20:23:44.603 INFO:teuthology.orchestra.run.smithi129.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-15T20:23:44.629 INFO:teuthology.orchestra.run.smithi129.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-09-15T20:23:44.645 INFO:teuthology.orchestra.run.smithi129.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-09-15T20:23:44.679 INFO:teuthology.orchestra.run.smithi129.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-15T20:23:44.684 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:23:44.684 INFO:teuthology.orchestra.run.smithi129.stdout:Total 9.5 MB/s | 31 MB 00:03 2024-09-15T20:23:44.837 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:23:44.877 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:23:44.877 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:23:45.342 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:23:45.342 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:23:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:23:46.740 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-15T20:23:46.773 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-15T20:23:46.801 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-15T20:23:46.871 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-15T20:23:46.942 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-15T20:23:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-15T20:23:47.066 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-15T20:23:47.138 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-15T20:23:47.194 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-15T20:23:47.238 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-15T20:23:47.268 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-15T20:23:47.303 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-15T20:23:47.336 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-15T20:23:47.376 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-15T20:23:47.411 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-15T20:23:47.451 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-15T20:23:47.499 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-15T20:23:47.531 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-15T20:23:47.604 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-15T20:23:47.742 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-15T20:23:47.823 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-15T20:23:47.877 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-15T20:23:47.924 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-15T20:23:47.989 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-15T20:23:48.046 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-15T20:23:48.230 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-15T20:23:48.278 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-15T20:23:48.356 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-15T20:23:48.446 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-15T20:23:48.495 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-15T20:23:48.629 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-15T20:23:48.797 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 32/37 2024-09-15T20:23:48.822 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-15T20:23:48.849 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-15T20:23:48.849 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:23:48.849 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T20:23:48.849 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T20:23:48.849 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T20:23:48.849 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:50.306 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-15T20:23:50.333 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-15T20:23:50.333 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:23:50.333 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T20:23:50.334 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T20:23:50.334 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T20:23:50.334 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:50.790 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-15T20:23:50.815 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-15T20:23:50.815 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:23:50.815 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T20:23:50.815 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T20:23:50.815 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T20:23:50.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:51.041 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-15T20:23:51.065 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-15T20:23:51.065 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:23:51.065 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T20:23:51.065 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T20:23:51.065 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T20:23:51.065 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:51.129 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-15T20:23:52.902 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-15T20:23:52.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/37 2024-09-15T20:23:52.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/37 2024-09-15T20:23:52.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/37 2024-09-15T20:23:52.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/37 2024-09-15T20:23:52.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 6/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-15T20:23:52.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-15T20:23:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-15T20:23:52.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-15T20:23:52.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-15T20:23:52.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-15T20:23:52.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-15T20:23:52.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-15T20:23:53.885 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-15T20:23:53.885 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T20:23:53.886 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T20:23:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T20:23:53.888 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T20:23:53.889 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T20:23:53.889 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:53.889 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:23:54.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base 2024-09-15T20:23:54.839 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:23:54.928 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:23:54.970 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:23:54.971 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:23:54.971 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:23:55.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install cephadm 2024-09-15T20:23:55.592 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:23:55.725 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 784 k 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:23:55.726 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:55.727 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 784 k 2024-09-15T20:23:55.727 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 791 k 2024-09-15T20:23:55.727 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:23:56.333 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm-19.3.0-4930.gfc3e0bff.el9.noarch.rpm 1.3 MB/s | 784 kB 00:00 2024-09-15T20:23:56.334 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:23:56.334 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.3 MB/s | 784 kB 00:00 2024-09-15T20:23:56.334 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:23:56.339 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:23:56.339 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:23:56.345 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:23:56.346 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:23:56.419 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:23:56.920 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:23:57.013 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:23:57.370 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:23:57.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:23:57.628 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:57.628 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:23:57.628 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:23:57.628 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:57.629 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:23:57.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T20:23:58.419 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:59 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:23:58.557 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:23:58.557 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:58.557 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 145 k 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 145 k 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 432 k 2024-09-15T20:23:58.558 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:23:58.759 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-19.3.0-4930.gfc3e0b 723 kB/s | 145 kB 00:00 2024-09-15T20:23:58.759 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:23:58.759 INFO:teuthology.orchestra.run.smithi129.stdout:Total 718 kB/s | 145 kB 00:00 2024-09-15T20:23:58.760 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:23:58.767 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:23:58.767 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:23:58.824 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:23:58.824 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:23:58.970 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:23:59.043 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T20:23:59.068 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T20:23:59.068 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:23:59.068 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T20:23:59.068 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:59.961 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T20:23:59.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:59.961 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:23:59.961 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:23:59.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:23:59.962 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:00.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr 2024-09-15T20:24:00.775 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:01 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:00.869 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:00.913 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:00.914 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:00.914 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:00.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T20:24:01.532 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:02 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:01.666 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:01.667 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:01.667 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:24:01.667 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:01.667 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:01.667 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 14 M 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 34 k 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 16 k 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T20:24:01.668 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T20:24:01.669 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T20:24:01.669 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:01.669 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:01.669 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:01.669 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-09-15T20:24:01.669 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:01.670 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 19 M 2024-09-15T20:24:01.670 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 159 M 2024-09-15T20:24:01.670 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:02.163 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4930.gfc3 96 kB/s | 16 kB 00:00 2024-09-15T20:24:02.188 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4930.gfc 174 kB/s | 34 kB 00:00 2024-09-15T20:24:02.347 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-15T20:24:02.414 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-15T20:24:02.472 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-15T20:24:02.498 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-09-15T20:24:02.573 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-15T20:24:02.598 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-15T20:24:02.624 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-15T20:24:02.649 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-15T20:24:02.682 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.5 MB/s | 863 kB 00:00 2024-09-15T20:24:03.650 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4930.gfc3e0b 8.5 MB/s | 14 MB 00:01 2024-09-15T20:24:03.653 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:03.653 INFO:teuthology.orchestra.run.smithi129.stdout:Total 9.7 MB/s | 19 MB 00:01 2024-09-15T20:24:03.700 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:03.719 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:03.719 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:03.987 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:03.988 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:04.578 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:04.658 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-15T20:24:04.702 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-15T20:24:04.749 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-15T20:24:04.788 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-15T20:24:04.875 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-15T20:24:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-15T20:24:05.092 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-15T20:24:05.156 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-15T20:24:05.197 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-15T20:24:05.229 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 10/12 2024-09-15T20:24:08.419 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 11/12 2024-09-15T20:24:08.475 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-15T20:24:10.508 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 1/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 2/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 3/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-15T20:24:10.509 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-15T20:24:10.510 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-15T20:24:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-15T20:24:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T20:24:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T20:24:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:10.837 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:11.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T20:24:11.690 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:12 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:11.833 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 7.4 M 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T20:24:11.834 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-15T20:24:11.835 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-09-15T20:24:11.836 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:11.836 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 38 M 2024-09-15T20:24:11.836 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 212 M 2024-09-15T20:24:11.836 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:12.726 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 296 kB/s | 191 kB 00:00 2024-09-15T20:24:12.834 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-09-15T20:24:12.951 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 921 kB/s | 801 kB 00:00 2024-09-15T20:24:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-09-15T20:24:13.094 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 128 kB/s | 18 kB 00:00 2024-09-15T20:24:13.136 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 510 kB/s | 38 kB 00:00 2024-09-15T20:24:13.219 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-15T20:24:13.295 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 25 MB/s | 4.9 MB 00:00 2024-09-15T20:24:13.413 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): ceph-mgr-diskprediction-local-19.3.0-49 5.6 MB/s | 7.4 MB 00:01 2024-09-15T20:24:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.7 MB/s | 250 kB 00:00 2024-09-15T20:24:13.512 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-15T20:24:13.837 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-09-15T20:24:13.838 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:13.839 INFO:teuthology.orchestra.run.smithi129.stdout:Total 19 MB/s | 38 MB 00:02 2024-09-15T20:24:14.195 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:14.221 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:14.221 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:14.674 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:14.674 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:15.433 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:15.515 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T20:24:15.552 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T20:24:15.655 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T20:24:15.687 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T20:24:15.871 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T20:24:15.902 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T20:24:15.984 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T20:24:16.654 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T20:24:16.784 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T20:24:18.056 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T20:24:19.121 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T20:24:19.154 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-15T20:24:21.732 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 1/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T20:24:21.733 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T20:24:22.151 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T20:24:22.151 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:22.151 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:22.152 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:24:22.152 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T20:24:22.152 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T20:24:22.152 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T20:24:22.152 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T20:24:22.152 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T20:24:22.153 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T20:24:22.153 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T20:24:22.153 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T20:24:22.153 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T20:24:22.153 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T20:24:22.153 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T20:24:22.154 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:22.154 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:22.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-rook 2024-09-15T20:24:23.132 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:24 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:23.266 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:23.267 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:23.267 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 49 k 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T20:24:23.268 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout:Install 13 Packages 2024-09-15T20:24:23.269 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:23.270 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.2 M 2024-09-15T20:24:23.270 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-09-15T20:24:23.270 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:23.438 INFO:teuthology.orchestra.run.smithi129.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 219 kB/s | 19 kB 00:00 2024-09-15T20:24:23.455 INFO:teuthology.orchestra.run.smithi129.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 246 kB/s | 26 kB 00:00 2024-09-15T20:24:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-15T20:24:23.573 INFO:teuthology.orchestra.run.smithi129.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-15T20:24:23.598 INFO:teuthology.orchestra.run.smithi129.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.7 MB/s | 279 kB 00:00 2024-09-15T20:24:23.615 INFO:teuthology.orchestra.run.smithi129.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 54 kB 00:00 2024-09-15T20:24:23.648 INFO:teuthology.orchestra.run.smithi129.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 641 kB/s | 32 kB 00:00 2024-09-15T20:24:23.666 INFO:teuthology.orchestra.run.smithi129.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 281 kB/s | 14 kB 00:00 2024-09-15T20:24:23.691 INFO:teuthology.orchestra.run.smithi129.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 5.3 MB/s | 220 kB 00:00 2024-09-15T20:24:23.716 INFO:teuthology.orchestra.run.smithi129.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T20:24:23.741 INFO:teuthology.orchestra.run.smithi129.stdout:(11/13): ceph-mgr-rook-19.3.0-4930.gfc3e0bff.el 126 kB/s | 49 kB 00:00 2024-09-15T20:24:23.766 INFO:teuthology.orchestra.run.smithi129.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-15T20:24:23.833 INFO:teuthology.orchestra.run.smithi129.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-15T20:24:23.836 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:23.836 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.9 MB/s | 2.2 MB 00:00 2024-09-15T20:24:23.901 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:23.916 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:23.916 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:24.006 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:24.007 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:24.504 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:24.662 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T20:24:24.718 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T20:24:24.783 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T20:24:24.819 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T20:24:24.861 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T20:24:24.964 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T20:24:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T20:24:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T20:24:25.844 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T20:24:25.902 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T20:24:25.936 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T20:24:25.984 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T20:24:26.006 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-15T20:24:26.811 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T20:24:26.812 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T20:24:26.813 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T20:24:27.142 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T20:24:27.143 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T20:24:27.143 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T20:24:27.143 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T20:24:27.143 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T20:24:27.143 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T20:24:27.143 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:27.143 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:27.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T20:24:27.989 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:28.124 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:28.125 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:28.125 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:24:28.125 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:28.125 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 154 k 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-09-15T20:24:28.126 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:28.127 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 7.0 M 2024-09-15T20:24:28.127 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 33 M 2024-09-15T20:24:28.127 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:28.563 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-15T20:24:28.588 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4930.gfc3e0bff.e 631 kB/s | 154 kB 00:00 2024-09-15T20:24:28.639 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-15T20:24:28.664 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-15T20:24:28.689 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-15T20:24:30.872 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.4 MB/s | 6.0 MB 00:02 2024-09-15T20:24:30.875 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:30.876 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.6 MB/s | 7.0 MB 00:02 2024-09-15T20:24:30.950 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:30.962 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:30.962 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:31.063 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:31.063 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:31.400 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:31.581 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T20:24:31.632 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T20:24:32.148 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T20:24:32.236 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T20:24:32.302 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T20:24:32.328 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-15T20:24:33.046 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-15T20:24:33.046 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/6 2024-09-15T20:24:33.046 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T20:24:33.046 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T20:24:33.046 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T20:24:33.046 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T20:24:33.330 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T20:24:33.330 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:33.330 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:33.330 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:24:33.330 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T20:24:33.330 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T20:24:33.331 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T20:24:33.331 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T20:24:33.331 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T20:24:33.331 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:33.331 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:33.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse 2024-09-15T20:24:34.144 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:34.278 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:34.278 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:24:34.278 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:34.278 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 900 k 2024-09-15T20:24:34.279 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-15T20:24:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T20:24:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:34.279 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:34.279 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:34.279 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-09-15T20:24:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:34.280 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 980 k 2024-09-15T20:24:34.280 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.9 M 2024-09-15T20:24:34.280 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:34.516 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 586 kB/s | 80 kB 00:00 2024-09-15T20:24:34.975 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-fuse-19.3.0-4930.gfc3e0bff.el9.x86_ 1.5 MB/s | 900 kB 00:00 2024-09-15T20:24:34.975 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:34.976 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.4 MB/s | 980 kB 00:00 2024-09-15T20:24:34.988 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:35.011 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:35.011 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:35.091 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:35.091 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:35.285 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:35.436 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T20:24:35.507 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-15T20:24:36.180 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-15T20:24:36.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/2 2024-09-15T20:24:36.457 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T20:24:36.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:36.458 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:36.458 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T20:24:36.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:36.458 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:36.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-volume 2024-09-15T20:24:37.302 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:37.444 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 285 k 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:37.445 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:24:37.446 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:37.446 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 285 k 2024-09-15T20:24:37.446 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.3 M 2024-09-15T20:24:37.446 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:37.656 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-volume-19.3.0-4930.gfc3e0bff.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-15T20:24:37.657 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:37.657 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-15T20:24:37.657 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:37.665 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:37.665 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:37.692 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:37.692 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:37.983 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:24:37.999 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:24:37.999 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:24:37.999 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T20:24:37.999 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:39.064 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:24:39.065 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:39.065 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:39.065 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:24:39.065 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:39.065 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:39.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel 2024-09-15T20:24:39.968 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:40.111 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:40.112 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:40.112 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 127 k 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 127 k 2024-09-15T20:24:40.113 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 456 k 2024-09-15T20:24:40.114 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:40.330 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-19.3.0-4930.gfc3e0bff.el9.x86_64 585 kB/s | 127 kB 00:00 2024-09-15T20:24:40.331 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:40.331 INFO:teuthology.orchestra.run.smithi129.stdout:Total 581 kB/s | 127 kB 00:00 2024-09-15T20:24:40.331 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:40.336 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:40.336 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:40.402 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:40.402 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:40.558 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:40.694 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:41.170 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:41.441 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:41.441 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:41.441 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:41.441 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:24:41.441 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:41.441 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:41.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2 2024-09-15T20:24:42.323 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:42.429 INFO:teuthology.orchestra.run.smithi129.stdout:Package libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:42.474 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:42.475 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:42.475 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:42.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs-devel 2024-09-15T20:24:43.119 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:43.252 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:43.252 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 32 k 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 32 k 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 142 k 2024-09-15T20:24:43.253 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:43.569 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs-devel-19.3.0-4930.gfc3e0bff.el9.x86_6 101 kB/s | 32 kB 00:00 2024-09-15T20:24:43.569 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:43.570 INFO:teuthology.orchestra.run.smithi129.stdout:Total 100 kB/s | 32 kB 00:00 2024-09-15T20:24:43.570 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:43.573 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:43.573 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:43.594 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:43.595 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:43.682 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:43.808 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:44.246 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:44.496 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:44.497 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:44.497 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:44.497 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:24:44.497 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:44.497 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:44.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2 2024-09-15T20:24:45.261 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:45.350 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:45.394 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:45.395 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:45.395 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:45.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1 2024-09-15T20:24:46.032 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:46.124 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:46.167 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:46.168 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:46.168 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:46.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados 2024-09-15T20:24:46.800 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:46.891 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:46.934 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:46.935 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:46.935 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:47.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw 2024-09-15T20:24:47.585 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:48 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:47.675 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:47.719 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:47.720 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:47.720 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:47.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs 2024-09-15T20:24:48.364 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:49 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:48.455 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:48.497 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:48.498 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:48.498 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:48.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd 2024-09-15T20:24:49.122 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:50 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:49.212 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:24:49.256 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:49.257 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-15T20:24:49.257 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:49.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse 2024-09-15T20:24:49.905 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:50 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:50.038 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 85 k 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:24:50.039 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:50.040 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 85 k 2024-09-15T20:24:50.040 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 231 k 2024-09-15T20:24:50.040 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:50.339 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 285 kB/s | 85 kB 00:00 2024-09-15T20:24:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:Total 283 kB/s | 85 kB 00:00 2024-09-15T20:24:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:50.410 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:50.410 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:50.590 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:50.686 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:51.061 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:51.305 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:51.305 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:51.305 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:51.305 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:24:51.305 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:51.305 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:51.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror 2024-09-15T20:24:52.154 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:53 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:52.293 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:24:52.294 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:52.295 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.2 M 2024-09-15T20:24:52.295 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 14 M 2024-09-15T20:24:52.295 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:53.144 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 3.8 MB/s | 3.2 MB 00:00 2024-09-15T20:24:53.145 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:53.145 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.7 MB/s | 3.2 MB 00:00 2024-09-15T20:24:53.145 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:53.154 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:53.154 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:53.236 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:53.236 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:53.680 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:53.745 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:53.770 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:53.770 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:24:53.770 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T20:24:53.770 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T20:24:53.770 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T20:24:53.770 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:54.937 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:54.937 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:54.937 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:54.937 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:24:54.937 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:54.938 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:55.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd 2024-09-15T20:24:55.738 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:56 ago on Sun 15 Sep 2024 08:22:59 PM UTC. 2024-09-15T20:24:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-15T20:24:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:55.875 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-15T20:24:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-15T20:24:55.875 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 170 k 2024-09-15T20:24:55.875 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-15T20:24:55.876 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-15T20:24:55.876 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-15T20:24:55.876 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:55.876 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 170 k 2024-09-15T20:24:55.876 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 490 k 2024-09-15T20:24:55.876 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-15T20:24:56.083 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 822 kB/s | 170 kB 00:00 2024-09-15T20:24:56.084 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:24:56.084 INFO:teuthology.orchestra.run.smithi129.stdout:Total 817 kB/s | 170 kB 00:00 2024-09-15T20:24:56.084 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-15T20:24:56.091 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-15T20:24:56.092 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-15T20:24:56.156 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-15T20:24:56.156 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-15T20:24:56.310 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-15T20:24:56.422 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:56.884 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:57.153 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:24:57.153 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:57.153 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-15T20:24:57.153 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:24:57.153 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:57.153 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-15T20:24:57.422 DEBUG:teuthology.parallel:result is None 2024-09-15T20:24:57.423 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:24:57.577 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T20:24:57.598 INFO:teuthology.orchestra.run.smithi129.stdout:19.3.0-4930.gfc3e0bff.el9 2024-09-15T20:24:57.598 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930.gfc3e0bff.el9 2024-09-15T20:24:57.598 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930.gfc3e0bff is installed. 2024-09-15T20:24:57.600 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T20:24:57.600 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:57.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T20:24:57.663 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T20:24:57.664 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:57.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T20:24:57.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T20:24:57.793 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T20:24:57.793 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:57.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T20:24:57.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T20:24:57.922 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T20:24:57.922 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:57.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T20:24:57.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T20:24:58.050 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T20:24:58.130 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T20:24:58.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-09-15T20:24:58.155 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T20:24:58.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T20:24:58.220 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T20:24:58.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T20:24:58.292 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T20:24:58.292 INFO:tasks.ceph:config {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['Error', 'overall HEALTH_', '\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_NEARFULL\\)', '\\(SLOW_OPS\\)', '\\(TOO_FEW_PGS\\)', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T20:24:58.292 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604', 'branch': 'main', 'description': 'rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905604', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 2855, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'sleep_before_teardown': 0, 'subset': '17936/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7498493279982dfe87f00616198a5967475f6169', 'targets': {'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAMOovC30gXvL370lpa50lS5EQ/JX7oF8ii7aeE/RBbF+BDUVyfTd1pj6LBOVaiuFnjHeCX8Ayuo9xboStDQrjA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'create_rbd_pool': False, 'log-ignorelist': ['Error', 'overall HEALTH_', '\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_NEARFULL\\)', '\\(SLOW_OPS\\)', '\\(TOO_FEW_PGS\\)', 'Monitor daemon marked osd\\.[[:digit:]]+ down, but it is still running', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'cluster': 'ceph'}}, {'backfill_toofull': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_20:00:16', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T20:24:58.293 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T20:24:58.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T20:24:58.404 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:58.404 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T20:24:58.459 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T20:24:58.460 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-15 20:24:54.022760703 +0000 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-15 20:22:18.117874036 +0000 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-15 20:22:18.117874036 +0000 2024-09-15T20:24:58.515 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-15 20:22:18.117874036 +0000 2024-09-15T20:24:58.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T20:24:58.577 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-15T20:24:58.578 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-15T20:24:58.578 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000186361 s, 2.7 MB/s 2024-09-15T20:24:58.579 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T20:24:58.635 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-15 20:24:54.023760681 +0000 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-15 20:22:18.480866194 +0000 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-15 20:22:18.480866194 +0000 2024-09-15T20:24:58.690 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-15 20:22:18.480866194 +0000 2024-09-15T20:24:58.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T20:24:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-15T20:24:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-15T20:24:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000261932 s, 2.0 MB/s 2024-09-15T20:24:58.754 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T20:24:58.811 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-15 20:24:54.023760681 +0000 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-15 20:22:18.814858978 +0000 2024-09-15T20:24:58.866 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-15 20:22:18.814858978 +0000 2024-09-15T20:24:58.867 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-15 20:22:18.814858978 +0000 2024-09-15T20:24:58.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T20:24:58.929 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-15T20:24:58.929 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-15T20:24:58.929 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000227094 s, 2.3 MB/s 2024-09-15T20:24:58.930 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T20:24:58.987 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-09-15T20:24:59.042 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T20:24:59.042 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:24:59.042 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T20:24:59.042 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:24:59.043 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T20:24:59.043 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-15 20:24:54.023760681 +0000 2024-09-15T20:24:59.043 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-15 20:22:19.139851957 +0000 2024-09-15T20:24:59.043 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-15 20:22:19.139851957 +0000 2024-09-15T20:24:59.043 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-15 20:22:19.139851957 +0000 2024-09-15T20:24:59.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T20:24:59.106 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-15T20:24:59.106 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-15T20:24:59.106 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000234896 s, 2.2 MB/s 2024-09-15T20:24:59.107 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T20:24:59.164 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T20:24:59.164 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-15T20:24:59.164 INFO:tasks.ceph:Generating config... 2024-09-15T20:24:59.165 INFO:tasks.ceph:[osd] osd max pg log entries = 5 2024-09-15T20:24:59.165 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2024-09-15T20:24:59.165 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T20:24:59.165 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-15T20:24:59.165 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T20:24:59.165 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T20:24:59.166 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T20:24:59.167 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T20:24:59.167 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T20:24:59.167 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T20:24:59.167 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T20:24:59.167 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-15T20:24:59.167 INFO:tasks.ceph:[global] ms type = async 2024-09-15T20:24:59.167 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T20:24:59.167 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T20:24:59.167 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T20:24:59.167 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T20:24:59.167 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T20:24:59.167 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T20:24:59.167 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T20:24:59.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T20:24:59.252 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T20:24:59.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T20:24:59.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T20:24:59.361 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.129'), ('mon.b', '[v2:172.21.15.129:3301,v1:172.21.15.129:6790]'), ('mon.c', '[v2:172.21.15.129:3302,v1:172.21.15.129:6791]')] 2024-09-15T20:24:59.362 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790],[v2:172.21.15.129:3302,v1:172.21.15.129:6791]', 'mon election default strategy': 3, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-15T20:24:59.362 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:59.362 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T20:24:59.417 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.129 --addv b '[v2:172.21.15.129:3301,v1:172.21.15.129:6790]' --addv c '[v2:172.21.15.129:3302,v1:172.21.15.129:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:24:59.504 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid 54dc298d-7b27-4c9d-9204-d7c9a8f9d562 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = quincy 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:fsid 54dc298d-7b27-4c9d-9204-d7c9a8f9d562 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2024-09-15T20:24:59.503716+0000 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-09-15T20:24:59.503716+0000 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 17 (quincy) 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:election_strategy: 3 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:1: [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon.b 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:2: [v2:172.21.15.129:3302/0,v1:172.21.15.129:6791/0] mon.c 2024-09-15T20:24:59.505 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T20:24:59.507 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T20:24:59.521 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 54dc298d-7b27-4c9d-9204-d7c9a8f9d562... 2024-09-15T20:24:59.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T20:24:59.604 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-15T20:24:59.604 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-15T20:24:59.604 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T20:24:59.604 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-15T20:24:59.604 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-15T20:24:59.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T20:24:59.606 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,[v2:172.21.15.129:3301,v1:172.21.15.129:6790],[v2:172.21.15.129:3302,v1:172.21.15.129:6791]" 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: mon election default strategy = 3 2024-09-15T20:24:59.607 INFO:teuthology.orchestra.run.smithi129.stdout: ms type = async 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = 54dc298d-7b27-4c9d-9204-d7c9a8f9d562 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-15T20:24:59.608 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-15T20:24:59.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd max pg log entries = 5 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: osd min pg log entries = 5 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression algorithm = zlib 2024-09-15T20:24:59.610 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression mode = aggressive 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore write v2 random = True 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore zero block detection = True 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 20 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 20 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 10 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify cached snaps = True 2024-09-15T20:24:59.611 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify missing on start = True 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock override recovery settings = True 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue = debug_random 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue cut off = debug_random 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-15T20:24:59.612 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-15T20:24:59.613 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.614 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: mon scrub interval = 300 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-15T20:24:59.615 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-15T20:24:59.616 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-15T20:24:59.616 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T20:24:59.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T20:24:59.712 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T20:24:59.712 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:59.713 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T20:24:59.728 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:59.728 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T20:24:59.783 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:24:59.783 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:59.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T20:24:59.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T20:24:59.858 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:24:59.858 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:24:59.913 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T20:24:59.913 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T20:24:59.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T20:25:00.003 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T20:25:00.006 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T20:25:00.006 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T20:25:00.006 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T20:25:00.006 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-15T20:25:00.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T20:25:00.031 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T20:25:00.032 INFO:tasks.ceph:role: osd.0 2024-09-15T20:25:00.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:25:00.032 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T20:25:00.104 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:25:00.104 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:25:00.104 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:25:00.104 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T20:25:00.104 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:25:00.104 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:25:00.105 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:25:00.105 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T20:25:00.105 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:25:00.105 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:25:00.604 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-15T20:25:00.606 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-15T20:25:00.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T20:25:00.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T20:25:00.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T20:25:00.822 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T20:25:00.823 INFO:tasks.ceph:role: osd.1 2024-09-15T20:25:00.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:25:00.823 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T20:25:00.895 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:25:00.895 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:25:00.895 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:25:00.895 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T20:25:00.895 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:25:00.895 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:25:00.895 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:25:00.896 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T20:25:00.896 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:25:00.896 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:25:01.357 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-15T20:25:01.360 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-15T20:25:01.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T20:25:01.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T20:25:01.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T20:25:01.538 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T20:25:01.539 INFO:tasks.ceph:role: osd.2 2024-09-15T20:25:01.539 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:25:01.539 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T20:25:01.611 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:25:01.611 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:25:01.611 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:25:01.611 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T20:25:01.611 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:25:01.611 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:25:01.611 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:25:01.612 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T20:25:01.612 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:25:01.612 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:25:02.092 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-15T20:25:02.094 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-15T20:25:02.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T20:25:02.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T20:25:02.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T20:25:02.331 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T20:25:02.331 INFO:tasks.ceph:role: osd.3 2024-09-15T20:25:02.331 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:25:02.331 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T20:25:02.404 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:25:02.404 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:25:02.405 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:25:02.920 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-15T20:25:02.922 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-15T20:25:02.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-15T20:25:03.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T20:25:03.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:25:03.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:03.189+0000 7fd97c40b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T20:25:03.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:03.189+0000 7fd97c40b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T20:25:03.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:03.189+0000 7fd97c40b640 -1 bdev(0x5625b3b5f800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-15T20:25:03.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:03.190+0000 7fd97c40b640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T20:25:06.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T20:25:06.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:25:06.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:06.703+0000 7f7e760b0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T20:25:06.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:06.704+0000 7f7e760b0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T20:25:06.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:06.704+0000 7f7e760b0640 -1 bdev(0x55a8e3a1b800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-15T20:25:06.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:06.704+0000 7f7e760b0640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T20:25:09.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T20:25:09.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:25:09.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:09.978+0000 7f101ea0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T20:25:09.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:09.979+0000 7f101ea0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T20:25:09.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:09.979+0000 7f101ea0f640 -1 bdev(0x562e4cad9800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-15T20:25:09.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:09.979+0000 7f101ea0f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T20:25:13.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T20:25:13.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:25:13.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:13.760+0000 7fa170fb2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T20:25:13.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:13.761+0000 7fa170fb2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T20:25:13.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:13.761+0000 7fa170fb2640 -1 bdev(0x559c0e1af800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-15T20:25:13.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-15T20:25:13.761+0000 7fa170fb2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T20:25:16.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T20:25:16.703 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T20:25:16.703 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:16.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T20:25:16.771 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:16.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T20:25:16.839 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:16.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T20:25:16.905 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:16.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T20:25:16.971 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:16.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T20:25:17.037 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T20:25:17.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T20:25:17.101 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-09-15T20:25:17.101 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCcQudm4pAsABAA8geHf5bjmlxy2JR6L4XrIA== 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCfQudmsepYCxAArMELFh5ZJ+aDpJArUMsNtA== 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCiQudm52L+KRAAeKI44aYk9hYlBORQp7Rz5Q== 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQClQudmTTRlOhAAWUMaS14ULgCmWHFV2I0frA== 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2024-09-15T20:25:17.102 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCpQudmD99lLRAAc8wv/JHCvnM8yDREHtdQVQ== 2024-09-15T20:25:17.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T20:25:17.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:25:17.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:25:17.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:25:17.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:25:17.398 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T20:25:17.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T20:25:17.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T20:25:17.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T20:25:17.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T20:25:17.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T20:25:17.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T20:25:17.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T20:25:17.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T20:25:17.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T20:25:17.914 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:25:17.969 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T20:25:17.969 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T20:25:17.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T20:25:18.011 INFO:tasks.ceph.mon.a:Started 2024-09-15T20:25:18.011 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T20:25:18.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T20:25:18.013 INFO:tasks.ceph.mon.b:Started 2024-09-15T20:25:18.013 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T20:25:18.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T20:25:18.016 INFO:tasks.ceph.mon.c:Started 2024-09-15T20:25:18.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2024-09-15T20:25:23.246 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T20:25:23.247 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T20:25:23.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T20:25:23.249 INFO:tasks.ceph.mgr.x:Started 2024-09-15T20:25:23.249 DEBUG:tasks.ceph:set 0 configs 2024-09-15T20:25:23.250 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T20:25:23.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T20:25:23.430 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-09-15T20:25:23.442 INFO:tasks.ceph:check_enable_crimson: False 2024-09-15T20:25:23.442 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T20:25:23.442 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:23.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T20:25:23.467 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:23.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T20:25:23.531 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:23.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T20:25:23.597 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-15T20:25:23.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T20:25:23.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new a52846f6-f66b-4721-b2f4-c44999f2c728 0 2024-09-15T20:25:23.888 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-09-15T20:25:23.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new e3b9918e-a312-4729-bc3d-392d02ead18c 1 2024-09-15T20:25:24.078 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-15T20:25:24.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new a7c6897c-1a1d-4652-94b3-764fe468eeae 2 2024-09-15T20:25:24.276 INFO:teuthology.orchestra.run.smithi129.stdout:2 2024-09-15T20:25:24.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 4858f290-1285-44ae-8797-0c63c6dcbef8 3 2024-09-15T20:25:24.467 INFO:teuthology.orchestra.run.smithi129.stdout:3 2024-09-15T20:25:24.478 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T20:25:24.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T20:25:24.480 INFO:tasks.ceph.osd.0:Started 2024-09-15T20:25:24.481 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T20:25:24.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T20:25:24.483 INFO:tasks.ceph.osd.1:Started 2024-09-15T20:25:24.483 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T20:25:24.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T20:25:24.485 INFO:tasks.ceph.osd.2:Started 2024-09-15T20:25:24.486 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T20:25:24.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T20:25:24.488 INFO:tasks.ceph.osd.3:Started 2024-09-15T20:25:24.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:24.682 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:24.682 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":6,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:24.460612+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:24.693 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T20:25:24.694 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T20:25:25.096 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:25.256 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-15T20:25:25.256 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":6,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:24.460612+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:25.266 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-15T20:25:25.709 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:25:25.707+0000 7f21e8317640 -1 Falling back to public interface 2024-09-15T20:25:26.212 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-15T20:25:26.209+0000 7f7947a62640 -1 Falling back to public interface 2024-09-15T20:25:26.213 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-15T20:25:26.210+0000 7f8695d45640 -1 Falling back to public interface 2024-09-15T20:25:26.238 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-15T20:25:26.236+0000 7f28fd327640 -1 Falling back to public interface 2024-09-15T20:25:28.564 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-15T20:25:28.562+0000 7f407831b640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T20:25:28.565 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-15T20:25:28.563+0000 7f407831b640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T20:25:29.781 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:25:29.778+0000 7f21e8317640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:25:30.066 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-15T20:25:30.063+0000 7f28fd327640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:25:30.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-15T20:25:30.230+0000 7f8695d45640 -1 osd.3 0 log_to_monitors true 2024-09-15T20:25:30.534 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-15T20:25:30.531+0000 7f7947a62640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:25:31.583 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-15T20:25:31.580+0000 7f28f92b6640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T20:25:31.583 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-15T20:25:31.580+0000 7f79439f1640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T20:25:31.583 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:25:31.580+0000 7f21e42a6640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T20:25:31.589 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-15T20:25:31.586+0000 7f28f50dd640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:25:31.598 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:25:31.596+0000 7f21e00cd640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:25:31.603 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-15T20:25:31.600+0000 7f793f818640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:25:31.669 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:31.977 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-15T20:25:31.977 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":8,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:31.563822+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:31.988 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-15T20:25:34.233 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-15T20:25:34.230+0000 7f8691cd4640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T20:25:34.243 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-15T20:25:34.240+0000 7f8689af3640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:25:38.392 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:38.685 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-15T20:25:38.685 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":12,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:35.599362+0000","last_up_change":"2024-09-15T20:25:34.593028+0000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:25:33.574537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6809","nonce":4274430252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6811","nonce":4274430252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6815","nonce":4274430252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6813","nonce":4274430252}]},"public_addr":"172.21.15.129:6809/4274430252","cluster_addr":"172.21.15.129:6811/4274430252","heartbeat_back_addr":"172.21.15.129:6815/4274430252","heartbeat_front_addr":"172.21.15.129:6813/4274430252","state":["exists","up"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6801","nonce":2094586637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6803","nonce":2094586637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6807","nonce":2094586637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6805","nonce":2094586637}]},"public_addr":"172.21.15.129:6801/2094586637","cluster_addr":"172.21.15.129:6803/2094586637","heartbeat_back_addr":"172.21.15.129:6807/2094586637","heartbeat_front_addr":"172.21.15.129:6805/2094586637","state":["exists","up"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6825","nonce":3344078257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6827","nonce":3344078257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6831","nonce":3344078257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6829","nonce":3344078257}]},"public_addr":"172.21.15.129:6825/3344078257","cluster_addr":"172.21.15.129:6827/3344078257","heartbeat_back_addr":"172.21.15.129:6831/3344078257","heartbeat_front_addr":"172.21.15.129:6829/3344078257","state":["exists","up"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6817","nonce":435893452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6819","nonce":435893452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6823","nonce":435893452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6821","nonce":435893452}]},"public_addr":"172.21.15.129:6817/435893452","cluster_addr":"172.21.15.129:6819/435893452","heartbeat_back_addr":"172.21.15.129:6823/435893452","heartbeat_front_addr":"172.21.15.129:6821/435893452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.577650+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:30.816456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.038929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.251155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:38.696 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-15T20:25:38.697 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T20:25:38.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-15T20:25:38.698 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T20:25:38.983 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:38.995 INFO:teuthology.orchestra.run.smithi129.stdout:[{"version":3,"timestamp":"2024-09-15T20:25:34.232202+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"9482.142431"}]}] 2024-09-15T20:25:38.996 INFO:tasks.ceph_manager:config epoch is 3 2024-09-15T20:25:38.996 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T20:25:38.996 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T20:25:38.996 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T20:25:39.328 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3971229056},{"type":"v1","addr":"172.21.15.129:6833","nonce":3971229056}]},"active_addr":"172.21.15.129:6833/3971229056","active_change":"2024-09-15T20:25:27.552496+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2773509574}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2754047648}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":3664294158}]}]} 2024-09-15T20:25:39.341 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T20:25:39.341 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T20:25:39.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:39.629 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:39.629 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":12,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:35.599362+0000","last_up_change":"2024-09-15T20:25:34.593028+0000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:25:33.574537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6809","nonce":4274430252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6811","nonce":4274430252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6815","nonce":4274430252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6813","nonce":4274430252}]},"public_addr":"172.21.15.129:6809/4274430252","cluster_addr":"172.21.15.129:6811/4274430252","heartbeat_back_addr":"172.21.15.129:6815/4274430252","heartbeat_front_addr":"172.21.15.129:6813/4274430252","state":["exists","up"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6801","nonce":2094586637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6803","nonce":2094586637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6807","nonce":2094586637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6805","nonce":2094586637}]},"public_addr":"172.21.15.129:6801/2094586637","cluster_addr":"172.21.15.129:6803/2094586637","heartbeat_back_addr":"172.21.15.129:6807/2094586637","heartbeat_front_addr":"172.21.15.129:6805/2094586637","state":["exists","up"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6825","nonce":3344078257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6827","nonce":3344078257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6831","nonce":3344078257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6829","nonce":3344078257}]},"public_addr":"172.21.15.129:6825/3344078257","cluster_addr":"172.21.15.129:6827/3344078257","heartbeat_back_addr":"172.21.15.129:6831/3344078257","heartbeat_front_addr":"172.21.15.129:6829/3344078257","state":["exists","up"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6817","nonce":435893452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6819","nonce":435893452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6823","nonce":435893452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6821","nonce":435893452}]},"public_addr":"172.21.15.129:6817/435893452","cluster_addr":"172.21.15.129:6819/435893452","heartbeat_back_addr":"172.21.15.129:6823/435893452","heartbeat_front_addr":"172.21.15.129:6821/435893452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.577650+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:30.816456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.038929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.251155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:39.642 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T20:25:39.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:39.925 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:39.925 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":12,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:35.599362+0000","last_up_change":"2024-09-15T20:25:34.593028+0000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:25:33.574537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6809","nonce":4274430252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6811","nonce":4274430252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6815","nonce":4274430252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6813","nonce":4274430252}]},"public_addr":"172.21.15.129:6809/4274430252","cluster_addr":"172.21.15.129:6811/4274430252","heartbeat_back_addr":"172.21.15.129:6815/4274430252","heartbeat_front_addr":"172.21.15.129:6813/4274430252","state":["exists","up"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6801","nonce":2094586637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6803","nonce":2094586637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6807","nonce":2094586637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6805","nonce":2094586637}]},"public_addr":"172.21.15.129:6801/2094586637","cluster_addr":"172.21.15.129:6803/2094586637","heartbeat_back_addr":"172.21.15.129:6807/2094586637","heartbeat_front_addr":"172.21.15.129:6805/2094586637","state":["exists","up"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6825","nonce":3344078257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6827","nonce":3344078257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6831","nonce":3344078257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6829","nonce":3344078257}]},"public_addr":"172.21.15.129:6825/3344078257","cluster_addr":"172.21.15.129:6827/3344078257","heartbeat_back_addr":"172.21.15.129:6831/3344078257","heartbeat_front_addr":"172.21.15.129:6829/3344078257","state":["exists","up"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6817","nonce":435893452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6819","nonce":435893452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6823","nonce":435893452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6821","nonce":435893452}]},"public_addr":"172.21.15.129:6817/435893452","cluster_addr":"172.21.15.129:6819/435893452","heartbeat_back_addr":"172.21.15.129:6823/435893452","heartbeat_front_addr":"172.21.15.129:6821/435893452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.577650+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:30.816456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.038929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.251155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:39.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T20:25:39.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T20:25:39.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T20:25:39.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T20:25:40.084 INFO:teuthology.orchestra.run.smithi129.stdout:38654705669 2024-09-15T20:25:40.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:25:40.086 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-09-15T20:25:40.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:25:40.087 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-09-15T20:25:40.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:25:40.088 INFO:teuthology.orchestra.run.smithi129.stdout:47244640261 2024-09-15T20:25:40.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:25:40.432 INFO:teuthology.orchestra.run.smithi129.stdout:38654705667 2024-09-15T20:25:40.433 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-09-15T20:25:40.447 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.2 2024-09-15T20:25:40.448 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.1 2024-09-15T20:25:40.502 INFO:teuthology.orchestra.run.smithi129.stdout:38654705667 2024-09-15T20:25:40.509 INFO:teuthology.orchestra.run.smithi129.stdout:47244640260 2024-09-15T20:25:40.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.0 2024-09-15T20:25:40.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.3 2024-09-15T20:25:41.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:25:41.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:25:41.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:25:41.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:25:41.765 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-09-15T20:25:41.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2024-09-15T20:25:41.783 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:41.824 INFO:teuthology.orchestra.run.smithi129.stdout:38654705669 2024-09-15T20:25:41.837 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2024-09-15T20:25:41.837 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:41.863 INFO:teuthology.orchestra.run.smithi129.stdout:47244640262 2024-09-15T20:25:41.876 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640262 for osd.3 2024-09-15T20:25:41.876 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:41.883 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-09-15T20:25:41.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2024-09-15T20:25:41.896 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:41.896 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T20:25:41.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:25:42.216 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:42.216 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:25:42.228 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-15T20:25:41.560761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108588,"kb_used_data":464,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378772,"statfs":{"total":386547056640,"available":386435862528,"internally_reserved":0,"allocated":475136,"data_stored":767268,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000287"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":36,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-15T20:25:35.613221+0000","last_change":"2024-09-15T20:25:35.238908+0000","last_active":"2024-09-15T20:25:35.613221+0000","last_peered":"2024-09-15T20:25:35.613221+0000","last_clean":"2024-09-15T20:25:35.613221+0000","last_became_active":"2024-09-15T20:25:35.237763+0000","last_became_peered":"2024-09-15T20:25:35.237763+0000","last_unstale":"2024-09-15T20:25:35.613221+0000","last_undegraded":"2024-09-15T20:25:35.613221+0000","last_fullsized":"2024-09-15T20:25:35.613221+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:48.064107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":27721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":106496,"data_stored":27721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":131072,"data_stored":355913,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":355913,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:25:42.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:25:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:42.508 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:25:42.520 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-15T20:25:41.560761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108588,"kb_used_data":464,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378772,"statfs":{"total":386547056640,"available":386435862528,"internally_reserved":0,"allocated":475136,"data_stored":767268,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000287"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":36,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-15T20:25:35.613221+0000","last_change":"2024-09-15T20:25:35.238908+0000","last_active":"2024-09-15T20:25:35.613221+0000","last_peered":"2024-09-15T20:25:35.613221+0000","last_clean":"2024-09-15T20:25:35.613221+0000","last_became_active":"2024-09-15T20:25:35.237763+0000","last_became_peered":"2024-09-15T20:25:35.237763+0000","last_unstale":"2024-09-15T20:25:35.613221+0000","last_undegraded":"2024-09-15T20:25:35.613221+0000","last_fullsized":"2024-09-15T20:25:35.613221+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:48.064107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":106496,"data_stored":27721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":106496,"data_stored":27721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":131072,"data_stored":355913,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":355913,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:25:42.520 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T20:25:42.520 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T20:25:42.521 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T20:25:42.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T20:25:42.838 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:42.838 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T20:25:42.851 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T20:25:42.851 INFO:teuthology.run_tasks:Running task backfill_toofull... 2024-09-15T20:25:42.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:43.147 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:43.148 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":12,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:35.599362+0000","last_up_change":"2024-09-15T20:25:34.593028+0000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:25:33.574537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6809","nonce":4274430252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6811","nonce":4274430252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6815","nonce":4274430252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6813","nonce":4274430252}]},"public_addr":"172.21.15.129:6809/4274430252","cluster_addr":"172.21.15.129:6811/4274430252","heartbeat_back_addr":"172.21.15.129:6815/4274430252","heartbeat_front_addr":"172.21.15.129:6813/4274430252","state":["exists","up"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6801","nonce":2094586637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6803","nonce":2094586637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6807","nonce":2094586637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6805","nonce":2094586637}]},"public_addr":"172.21.15.129:6801/2094586637","cluster_addr":"172.21.15.129:6803/2094586637","heartbeat_back_addr":"172.21.15.129:6807/2094586637","heartbeat_front_addr":"172.21.15.129:6805/2094586637","state":["exists","up"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6825","nonce":3344078257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6827","nonce":3344078257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6831","nonce":3344078257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6829","nonce":3344078257}]},"public_addr":"172.21.15.129:6825/3344078257","cluster_addr":"172.21.15.129:6827/3344078257","heartbeat_back_addr":"172.21.15.129:6831/3344078257","heartbeat_front_addr":"172.21.15.129:6829/3344078257","state":["exists","up"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6817","nonce":435893452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6819","nonce":435893452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6823","nonce":435893452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6821","nonce":435893452}]},"public_addr":"172.21.15.129:6817/435893452","cluster_addr":"172.21.15.129:6819/435893452","heartbeat_back_addr":"172.21.15.129:6823/435893452","heartbeat_front_addr":"172.21.15.129:6821/435893452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.577650+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:30.816456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.038929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.251155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:43.161 INFO:tasks.backfill_toofull.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T20:25:33.574537+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-15T20:25:43.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T20:25:43.445 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-15T20:25:43.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set backfill_toofull k=2 m=1 crush-failure-domain=osd 2024-09-15T20:25:44.649 INFO:tasks.backfill_toofull.ceph_manager:creating pool_name unique_pool_0 2024-09-15T20:25:44.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 1 1 erasure backfill_toofull 2024-09-15T20:25:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'unique_pool_0' created 2024-09-15T20:25:46.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-15T20:25:47.642 INFO:teuthology.orchestra.run.smithi129.stderr:set pool 2 min_size to 2 2024-09-15T20:25:47.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-15T20:25:48.654 INFO:teuthology.orchestra.run.smithi129.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-15T20:25:49.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_autoscale_mode off 2024-09-15T20:25:50.658 INFO:teuthology.orchestra.run.smithi129.stderr:set pool 2 pg_autoscale_mode to off 2024-09-15T20:25:50.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T20:25:50.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T20:25:50.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T20:25:50.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T20:25:50.836 INFO:teuthology.orchestra.run.smithi129.stdout:38654705672 2024-09-15T20:25:50.836 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-15T20:25:50.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:25:50.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:25:50.839 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-15T20:25:50.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:25:50.841 INFO:teuthology.orchestra.run.smithi129.stdout:47244640265 2024-09-15T20:25:50.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:25:51.150 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-15T20:25:51.169 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705672 got 38654705671 for osd.1 2024-09-15T20:25:51.182 INFO:teuthology.orchestra.run.smithi129.stdout:38654705670 2024-09-15T20:25:51.189 INFO:teuthology.orchestra.run.smithi129.stdout:38654705670 2024-09-15T20:25:51.196 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705671 got 38654705670 for osd.2 2024-09-15T20:25:51.202 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705671 got 38654705670 for osd.0 2024-09-15T20:25:51.212 INFO:teuthology.orchestra.run.smithi129.stdout:47244640263 2024-09-15T20:25:51.225 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640265 got 47244640263 for osd.3 2024-09-15T20:25:52.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:25:52.197 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:25:52.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:25:52.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:25:52.498 INFO:teuthology.orchestra.run.smithi129.stdout:38654705672 2024-09-15T20:25:52.516 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705672 got 38654705672 for osd.1 2024-09-15T20:25:52.516 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:52.520 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-15T20:25:52.531 INFO:teuthology.orchestra.run.smithi129.stdout:38654705671 2024-09-15T20:25:52.534 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705671 got 38654705671 for osd.0 2024-09-15T20:25:52.534 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:52.545 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705671 got 38654705671 for osd.2 2024-09-15T20:25:52.545 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:52.572 INFO:teuthology.orchestra.run.smithi129.stdout:47244640265 2024-09-15T20:25:52.585 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640265 got 47244640265 for osd.3 2024-09-15T20:25:52.585 DEBUG:teuthology.parallel:result is None 2024-09-15T20:25:52.585 INFO:tasks.backfill_toofull.ceph_manager:waiting for clean 2024-09-15T20:25:52.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:25:52.904 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:52.904 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:25:52.917 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-15T20:25:51.562315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108752,"kb_used_data":632,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378608,"statfs":{"total":386547056640,"available":386435694592,"internally_reserved":0,"allocated":647168,"data_stored":856657,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.928671"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T20:25:50.658760+0000","last_change":"2024-09-15T20:25:47.649115+0000","last_active":"2024-09-15T20:25:50.658760+0000","last_peered":"2024-09-15T20:25:50.658760+0000","last_clean":"2024-09-15T20:25:50.658760+0000","last_became_active":"2024-09-15T20:25:47.648765+0000","last_became_peered":"2024-09-15T20:25:47.648765+0000","last_unstale":"2024-09-15T20:25:50.658760+0000","last_undegraded":"2024-09-15T20:25:50.658760+0000","last_fullsized":"2024-09-15T20:25:50.658760+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:24.230950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":45,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-15T20:25:48.652231+0000","last_change":"2024-09-15T20:25:35.238908+0000","last_active":"2024-09-15T20:25:48.652231+0000","last_peered":"2024-09-15T20:25:48.652231+0000","last_clean":"2024-09-15T20:25:48.652231+0000","last_became_active":"2024-09-15T20:25:35.237763+0000","last_became_peered":"2024-09-15T20:25:35.237763+0000","last_unstale":"2024-09-15T20:25:48.652231+0000","last_undegraded":"2024-09-15T20:25:48.652231+0000","last_fullsized":"2024-09-15T20:25:48.652231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:48.064107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":48981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":48981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":377173,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":381522,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:25:52.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:25:53.197 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:53.198 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:25:53.210 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-15T20:25:51.562315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108752,"kb_used_data":632,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378608,"statfs":{"total":386547056640,"available":386435694592,"internally_reserved":0,"allocated":647168,"data_stored":856657,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.928671"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T20:25:50.658760+0000","last_change":"2024-09-15T20:25:47.649115+0000","last_active":"2024-09-15T20:25:50.658760+0000","last_peered":"2024-09-15T20:25:50.658760+0000","last_clean":"2024-09-15T20:25:50.658760+0000","last_became_active":"2024-09-15T20:25:47.648765+0000","last_became_peered":"2024-09-15T20:25:47.648765+0000","last_unstale":"2024-09-15T20:25:50.658760+0000","last_undegraded":"2024-09-15T20:25:50.658760+0000","last_fullsized":"2024-09-15T20:25:50.658760+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:24.230950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":45,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-15T20:25:48.652231+0000","last_change":"2024-09-15T20:25:35.238908+0000","last_active":"2024-09-15T20:25:48.652231+0000","last_peered":"2024-09-15T20:25:48.652231+0000","last_clean":"2024-09-15T20:25:48.652231+0000","last_became_active":"2024-09-15T20:25:35.237763+0000","last_became_peered":"2024-09-15T20:25:35.237763+0000","last_unstale":"2024-09-15T20:25:48.652231+0000","last_undegraded":"2024-09-15T20:25:48.652231+0000","last_fullsized":"2024-09-15T20:25:48.652231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:48.064107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":48981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":48981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":377173,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":381522,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:25:53.210 INFO:tasks.backfill_toofull.ceph_manager:clean! 2024-09-15T20:25:53.211 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:25:53.498 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:53.498 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":18,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:25:50.652962+0000","last_up_change":"2024-09-15T20:25:34.593028+0000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:25:33.574537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2024-09-15T20:25:45.636932+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_toofull","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6809","nonce":4274430252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6811","nonce":4274430252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6815","nonce":4274430252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6813","nonce":4274430252}]},"public_addr":"172.21.15.129:6809/4274430252","cluster_addr":"172.21.15.129:6811/4274430252","heartbeat_back_addr":"172.21.15.129:6815/4274430252","heartbeat_front_addr":"172.21.15.129:6813/4274430252","state":["exists","up"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6801","nonce":2094586637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6803","nonce":2094586637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6807","nonce":2094586637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":2094586637},{"type":"v1","addr":"172.21.15.129:6805","nonce":2094586637}]},"public_addr":"172.21.15.129:6801/2094586637","cluster_addr":"172.21.15.129:6803/2094586637","heartbeat_back_addr":"172.21.15.129:6807/2094586637","heartbeat_front_addr":"172.21.15.129:6805/2094586637","state":["exists","up"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6825","nonce":3344078257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6827","nonce":3344078257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6831","nonce":3344078257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6829","nonce":3344078257}]},"public_addr":"172.21.15.129:6825/3344078257","cluster_addr":"172.21.15.129:6827/3344078257","heartbeat_back_addr":"172.21.15.129:6831/3344078257","heartbeat_front_addr":"172.21.15.129:6829/3344078257","state":["exists","up"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6817","nonce":435893452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6819","nonce":435893452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6823","nonce":435893452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6821","nonce":435893452}]},"public_addr":"172.21.15.129:6817/435893452","cluster_addr":"172.21.15.129:6819/435893452","heartbeat_back_addr":"172.21.15.129:6823/435893452","heartbeat_front_addr":"172.21.15.129:6821/435893452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.577650+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:30.816456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.038929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.251155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_toofull":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:25:53.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:25:53.794 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:25:53.794 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:25:53.806 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-15T20:25:53.562732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108752,"kb_used_data":632,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378608,"statfs":{"total":386547056640,"available":386435694592,"internally_reserved":0,"allocated":647168,"data_stored":856657,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001038"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T20:25:50.658760+0000","last_change":"2024-09-15T20:25:47.649115+0000","last_active":"2024-09-15T20:25:50.658760+0000","last_peered":"2024-09-15T20:25:50.658760+0000","last_clean":"2024-09-15T20:25:50.658760+0000","last_became_active":"2024-09-15T20:25:47.648765+0000","last_became_peered":"2024-09-15T20:25:47.648765+0000","last_unstale":"2024-09-15T20:25:50.658760+0000","last_undegraded":"2024-09-15T20:25:50.658760+0000","last_fullsized":"2024-09-15T20:25:50.658760+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:24.230950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":45,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-15T20:25:48.652231+0000","last_change":"2024-09-15T20:25:35.238908+0000","last_active":"2024-09-15T20:25:48.652231+0000","last_peered":"2024-09-15T20:25:48.652231+0000","last_clean":"2024-09-15T20:25:48.652231+0000","last_became_active":"2024-09-15T20:25:35.237763+0000","last_became_peered":"2024-09-15T20:25:35.237763+0000","last_unstale":"2024-09-15T20:25:48.652231+0000","last_undegraded":"2024-09-15T20:25:48.652231+0000","last_fullsized":"2024-09-15T20:25:48.652231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:48.064107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":48981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":48981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":377173,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":381522,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:25:53.806 DEBUG:tasks.backfill_toofull:acting=[3, 1, 2] 2024-09-15T20:25:53.806 DEBUG:tasks.backfill_toofull:write some data 2024-09-15T20:25:53.806 INFO:tasks.util.rados:rados -p unique_pool_0 bench 120 write --no-cleanup 2024-09-15T20:25:53.807 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 120 write --no-cleanup 2024-09-15T20:25:53.854 INFO:teuthology.orchestra.run.smithi129.stdout:hints = 1 2024-09-15T20:25:53.856 INFO:teuthology.orchestra.run.smithi129.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 120 seconds or 0 objects 2024-09-15T20:25:53.856 INFO:teuthology.orchestra.run.smithi129.stdout:Object prefix: benchmark_data_smithi129_38846 2024-09-15T20:25:53.881 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:25:53.882 INFO:teuthology.orchestra.run.smithi129.stdout: 0 0 0 0 0 0 - 0 2024-09-15T20:25:54.882 INFO:teuthology.orchestra.run.smithi129.stdout: 1 16 90 74 295.984 296 0.196772 0.194959 2024-09-15T20:25:55.882 INFO:teuthology.orchestra.run.smithi129.stdout: 2 16 169 153 305.98 316 0.206307 0.198757 2024-09-15T20:25:56.882 INFO:teuthology.orchestra.run.smithi129.stdout: 3 16 245 229 305.312 304 0.215568 0.201113 2024-09-15T20:25:57.882 INFO:teuthology.orchestra.run.smithi129.stdout: 4 16 322 306 305.978 308 0.216513 0.203956 2024-09-15T20:25:58.882 INFO:teuthology.orchestra.run.smithi129.stdout: 5 16 398 382 305.577 304 0.208958 0.204613 2024-09-15T20:25:59.882 INFO:teuthology.orchestra.run.smithi129.stdout: 6 16 475 459 305.977 308 0.201172 0.205747 2024-09-15T20:26:00.882 INFO:teuthology.orchestra.run.smithi129.stdout: 7 16 550 534 305.12 300 0.218589 0.20629 2024-09-15T20:26:01.882 INFO:teuthology.orchestra.run.smithi129.stdout: 8 16 626 610 304.977 304 0.21409 0.207223 2024-09-15T20:26:02.882 INFO:teuthology.orchestra.run.smithi129.stdout: 9 16 701 685 304.421 300 0.213001 0.207624 2024-09-15T20:26:03.882 INFO:teuthology.orchestra.run.smithi129.stdout: 10 16 778 762 304.777 308 0.193672 0.207856 2024-09-15T20:26:04.882 INFO:teuthology.orchestra.run.smithi129.stdout: 11 16 856 840 305.431 312 0.179962 0.207759 2024-09-15T20:26:05.883 INFO:teuthology.orchestra.run.smithi129.stdout: 12 16 931 915 304.978 300 0.197561 0.208088 2024-09-15T20:26:06.883 INFO:teuthology.orchestra.run.smithi129.stdout: 13 16 1006 990 304.593 300 0.230591 0.208441 2024-09-15T20:26:07.884 INFO:teuthology.orchestra.run.smithi129.stdout: 14 16 1083 1067 304.835 308 0.220315 0.208536 2024-09-15T20:26:08.883 INFO:teuthology.orchestra.run.smithi129.stdout: 15 16 1157 1141 304.244 296 0.219837 0.208845 2024-09-15T20:26:09.883 INFO:teuthology.orchestra.run.smithi129.stdout: 16 16 1231 1215 303.728 296 0.209541 0.209335 2024-09-15T20:26:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: 17 16 1305 1289 303.271 296 0.214957 0.209645 2024-09-15T20:26:11.883 INFO:teuthology.orchestra.run.smithi129.stdout: 18 16 1379 1363 302.866 296 0.222479 0.20991 2024-09-15T20:26:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: 19 16 1455 1439 302.925 304 0.192389 0.210124 2024-09-15T20:26:13.883 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:26:13.882430+0000 min lat: 0.0616332 max lat: 0.253704 avg lat: 0.210668 2024-09-15T20:26:13.883 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:26:13.884 INFO:teuthology.orchestra.run.smithi129.stdout: 20 16 1527 1511 302.177 288 0.211641 0.210668 2024-09-15T20:26:14.883 INFO:teuthology.orchestra.run.smithi129.stdout: 21 16 1600 1584 301.691 292 0.213385 0.210984 2024-09-15T20:26:15.883 INFO:teuthology.orchestra.run.smithi129.stdout: 22 16 1675 1659 301.613 300 0.198901 0.211068 2024-09-15T20:26:16.883 INFO:teuthology.orchestra.run.smithi129.stdout: 23 16 1753 1737 302.064 312 0.198273 0.211107 2024-09-15T20:26:17.884 INFO:teuthology.orchestra.run.smithi129.stdout: 24 16 1827 1811 301.81 296 0.22242 0.211165 2024-09-15T20:26:18.884 INFO:teuthology.orchestra.run.smithi129.stdout: 25 16 1904 1888 302.057 308 0.190049 0.211115 2024-09-15T20:26:19.884 INFO:teuthology.orchestra.run.smithi129.stdout: 26 16 1978 1962 301.823 296 0.204296 0.211196 2024-09-15T20:26:20.884 INFO:teuthology.orchestra.run.smithi129.stdout: 27 16 2054 2038 301.902 304 0.210495 0.211146 2024-09-15T20:26:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: 28 16 2130 2114 301.977 304 0.213448 0.211222 2024-09-15T20:26:22.884 INFO:teuthology.orchestra.run.smithi129.stdout: 29 16 2205 2189 301.908 300 0.204305 0.211175 2024-09-15T20:26:23.884 INFO:teuthology.orchestra.run.smithi129.stdout: 30 16 2281 2265 301.977 304 0.203189 0.21123 2024-09-15T20:26:24.884 INFO:teuthology.orchestra.run.smithi129.stdout: 31 16 2355 2339 301.783 296 0.210294 0.211266 2024-09-15T20:26:25.884 INFO:teuthology.orchestra.run.smithi129.stdout: 32 16 2432 2416 301.976 308 0.190076 0.211346 2024-09-15T20:26:26.884 INFO:teuthology.orchestra.run.smithi129.stdout: 33 16 2503 2487 301.431 284 0.223106 0.211549 2024-09-15T20:26:27.884 INFO:teuthology.orchestra.run.smithi129.stdout: 34 16 2578 2562 301.388 300 0.213375 0.211592 2024-09-15T20:26:28.884 INFO:teuthology.orchestra.run.smithi129.stdout: 35 16 2656 2640 301.691 312 0.20698 0.211493 2024-09-15T20:26:29.885 INFO:teuthology.orchestra.run.smithi129.stdout: 36 16 2732 2716 301.754 304 0.207031 0.211466 2024-09-15T20:26:30.885 INFO:teuthology.orchestra.run.smithi129.stdout: 37 16 2808 2792 301.814 304 0.200555 0.211445 2024-09-15T20:26:31.885 INFO:teuthology.orchestra.run.smithi129.stdout: 38 16 2884 2868 301.871 304 0.213348 0.21142 2024-09-15T20:26:32.885 INFO:teuthology.orchestra.run.smithi129.stdout: 39 16 2959 2943 301.823 300 0.236075 0.211467 2024-09-15T20:26:33.885 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:26:33.884036+0000 min lat: 0.0616332 max lat: 0.264059 avg lat: 0.211559 2024-09-15T20:26:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:26:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: 40 16 3032 3016 301.577 292 0.233164 0.211559 2024-09-15T20:26:34.885 INFO:teuthology.orchestra.run.smithi129.stdout: 41 16 3108 3092 301.635 304 0.189364 0.211625 2024-09-15T20:26:35.885 INFO:teuthology.orchestra.run.smithi129.stdout: 42 16 3183 3167 301.595 300 0.212817 0.211629 2024-09-15T20:26:36.885 INFO:teuthology.orchestra.run.smithi129.stdout: 43 16 3257 3241 301.465 296 0.205409 0.211759 2024-09-15T20:26:37.885 INFO:teuthology.orchestra.run.smithi129.stdout: 44 16 3333 3317 301.522 304 0.211083 0.211734 2024-09-15T20:26:38.885 INFO:teuthology.orchestra.run.smithi129.stdout: 45 16 3408 3392 301.487 300 0.211397 0.211818 2024-09-15T20:26:39.886 INFO:teuthology.orchestra.run.smithi129.stdout: 46 16 3483 3467 301.453 300 0.211696 0.211794 2024-09-15T20:26:40.886 INFO:teuthology.orchestra.run.smithi129.stdout: 47 16 3558 3542 301.422 300 0.215455 0.211821 2024-09-15T20:26:41.886 INFO:teuthology.orchestra.run.smithi129.stdout: 48 16 3634 3618 301.475 304 0.197415 0.211845 2024-09-15T20:26:42.886 INFO:teuthology.orchestra.run.smithi129.stdout: 49 16 3708 3692 301.363 296 0.198612 0.211975 2024-09-15T20:26:43.886 INFO:teuthology.orchestra.run.smithi129.stdout: 50 16 3783 3767 301.335 300 0.201307 0.211942 2024-09-15T20:26:44.886 INFO:teuthology.orchestra.run.smithi129.stdout: 51 16 3858 3842 301.308 300 0.198686 0.21201 2024-09-15T20:26:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: 52 16 3932 3916 301.206 296 0.207417 0.212036 2024-09-15T20:26:46.886 INFO:teuthology.orchestra.run.smithi129.stdout: 53 16 4003 3987 300.881 284 0.225049 0.212233 2024-09-15T20:26:47.886 INFO:teuthology.orchestra.run.smithi129.stdout: 54 16 4078 4062 300.864 300 0.204558 0.212287 2024-09-15T20:26:48.886 INFO:teuthology.orchestra.run.smithi129.stdout: 55 16 4153 4137 300.848 300 0.199669 0.212285 2024-09-15T20:26:49.886 INFO:teuthology.orchestra.run.smithi129.stdout: 56 16 4227 4211 300.761 296 0.207351 0.212404 2024-09-15T20:26:50.887 INFO:teuthology.orchestra.run.smithi129.stdout: 57 16 4302 4286 300.747 300 0.209684 0.212443 2024-09-15T20:26:51.886 INFO:teuthology.orchestra.run.smithi129.stdout: 58 16 4374 4358 300.527 288 0.211491 0.212546 2024-09-15T20:26:52.887 INFO:teuthology.orchestra.run.smithi129.stdout: 59 16 4447 4431 300.382 292 0.218425 0.21266 2024-09-15T20:26:53.887 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:26:53.885867+0000 min lat: 0.0616332 max lat: 0.266094 avg lat: 0.212755 2024-09-15T20:26:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:26:53.887 INFO:teuthology.orchestra.run.smithi129.stdout: 60 16 4521 4505 300.309 296 0.198853 0.212755 2024-09-15T20:26:54.887 INFO:teuthology.orchestra.run.smithi129.stdout: 61 16 4597 4581 300.369 304 0.193259 0.2127 2024-09-15T20:26:55.887 INFO:teuthology.orchestra.run.smithi129.stdout: 62 16 4671 4655 300.298 296 0.206174 0.212734 2024-09-15T20:26:56.887 INFO:teuthology.orchestra.run.smithi129.stdout: 63 16 4745 4729 300.229 296 0.205139 0.21278 2024-09-15T20:26:57.887 INFO:teuthology.orchestra.run.smithi129.stdout: 64 16 4820 4804 300.225 300 0.225157 0.212762 2024-09-15T20:26:58.887 INFO:teuthology.orchestra.run.smithi129.stdout: 65 16 4895 4879 300.221 300 0.197719 0.212813 2024-09-15T20:26:59.887 INFO:teuthology.orchestra.run.smithi129.stdout: 66 16 4970 4954 300.218 300 0.207659 0.212816 2024-09-15T20:27:00.887 INFO:teuthology.orchestra.run.smithi129.stdout: 67 16 5046 5030 300.274 304 0.198016 0.212848 2024-09-15T20:27:01.887 INFO:teuthology.orchestra.run.smithi129.stdout: 68 16 5120 5104 300.21 296 0.2167 0.21286 2024-09-15T20:27:02.888 INFO:teuthology.orchestra.run.smithi129.stdout: 69 16 5193 5177 300.091 292 0.21925 0.212893 2024-09-15T20:27:03.887 INFO:teuthology.orchestra.run.smithi129.stdout: 70 16 5269 5253 300.146 304 0.210615 0.212916 2024-09-15T20:27:04.888 INFO:teuthology.orchestra.run.smithi129.stdout: 71 16 5346 5330 300.257 308 0.208729 0.212846 2024-09-15T20:27:05.888 INFO:teuthology.orchestra.run.smithi129.stdout: 72 16 5421 5405 300.253 300 0.213553 0.212856 2024-09-15T20:27:06.888 INFO:teuthology.orchestra.run.smithi129.stdout: 73 16 5495 5479 300.194 296 0.21529 0.212889 2024-09-15T20:27:07.888 INFO:teuthology.orchestra.run.smithi129.stdout: 74 16 5568 5552 300.083 292 0.199178 0.212969 2024-09-15T20:27:08.888 INFO:teuthology.orchestra.run.smithi129.stdout: 75 16 5642 5626 300.028 296 0.210328 0.212987 2024-09-15T20:27:09.888 INFO:teuthology.orchestra.run.smithi129.stdout: 76 16 5716 5700 299.975 296 0.206687 0.213035 2024-09-15T20:27:10.888 INFO:teuthology.orchestra.run.smithi129.stdout: 77 16 5790 5774 299.923 296 0.212799 0.213086 2024-09-15T20:27:11.888 INFO:teuthology.orchestra.run.smithi129.stdout: 78 16 5865 5849 299.924 300 0.205156 0.213107 2024-09-15T20:27:12.888 INFO:teuthology.orchestra.run.smithi129.stdout: 79 16 5937 5921 299.773 288 0.198582 0.213207 2024-09-15T20:27:13.888 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:27:13.887566+0000 min lat: 0.0616332 max lat: 0.266094 avg lat: 0.213225 2024-09-15T20:27:13.889 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:27:13.889 INFO:teuthology.orchestra.run.smithi129.stdout: 80 16 6010 5994 299.675 292 0.222461 0.213225 2024-09-15T20:27:14.888 INFO:teuthology.orchestra.run.smithi129.stdout: 81 16 6085 6069 299.679 300 0.202457 0.21326 2024-09-15T20:27:15.888 INFO:teuthology.orchestra.run.smithi129.stdout: 82 16 6159 6143 299.634 296 0.210115 0.21333 2024-09-15T20:27:16.889 INFO:teuthology.orchestra.run.smithi129.stdout: 83 16 6233 6217 299.59 296 0.232595 0.213356 2024-09-15T20:27:17.889 INFO:teuthology.orchestra.run.smithi129.stdout: 84 16 6304 6288 299.404 284 0.225896 0.213464 2024-09-15T20:27:18.889 INFO:teuthology.orchestra.run.smithi129.stdout: 85 16 6379 6363 299.41 300 0.20618 0.213485 2024-09-15T20:27:19.889 INFO:teuthology.orchestra.run.smithi129.stdout: 86 16 6451 6435 299.277 288 0.21555 0.213537 2024-09-15T20:27:20.889 INFO:teuthology.orchestra.run.smithi129.stdout: 87 16 6527 6511 299.331 304 0.20983 0.213558 2024-09-15T20:27:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: 88 16 6600 6584 299.248 292 0.218216 0.213592 2024-09-15T20:27:22.889 INFO:teuthology.orchestra.run.smithi129.stdout: 89 16 6676 6660 299.301 304 0.223283 0.213567 2024-09-15T20:27:23.889 INFO:teuthology.orchestra.run.smithi129.stdout: 90 16 6748 6732 299.175 288 0.227452 0.213616 2024-09-15T20:27:24.889 INFO:teuthology.orchestra.run.smithi129.stdout: 91 16 6824 6808 299.228 304 0.195606 0.213654 2024-09-15T20:27:25.889 INFO:teuthology.orchestra.run.smithi129.stdout: 92 16 6896 6880 299.106 288 0.254396 0.213635 2024-09-15T20:27:26.889 INFO:teuthology.orchestra.run.smithi129.stdout: 93 16 6968 6952 298.986 288 0.211046 0.21381 2024-09-15T20:27:27.889 INFO:teuthology.orchestra.run.smithi129.stdout: 94 16 7041 7025 298.911 292 0.218996 0.213851 2024-09-15T20:27:28.890 INFO:teuthology.orchestra.run.smithi129.stdout: 95 16 7114 7098 298.838 292 0.21776 0.213887 2024-09-15T20:27:29.890 INFO:teuthology.orchestra.run.smithi129.stdout: 96 16 7190 7174 298.892 304 0.209048 0.213864 2024-09-15T20:27:30.890 INFO:teuthology.orchestra.run.smithi129.stdout: 97 16 7263 7247 298.821 292 0.216285 0.213939 2024-09-15T20:27:31.890 INFO:teuthology.orchestra.run.smithi129.stdout: 98 16 7337 7321 298.792 296 0.222521 0.213923 2024-09-15T20:27:32.890 INFO:teuthology.orchestra.run.smithi129.stdout: 99 16 7412 7396 298.804 300 0.216575 0.213945 2024-09-15T20:27:33.890 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:27:33.889176+0000 min lat: 0.0616332 max lat: 0.290218 avg lat: 0.213964 2024-09-15T20:27:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:27:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: 100 16 7487 7471 298.815 300 0.195616 0.213964 2024-09-15T20:27:34.890 INFO:teuthology.orchestra.run.smithi129.stdout: 101 16 7559 7543 298.708 288 0.224788 0.213983 2024-09-15T20:27:35.890 INFO:teuthology.orchestra.run.smithi129.stdout: 102 16 7634 7618 298.72 300 0.217996 0.214014 2024-09-15T20:27:36.890 INFO:teuthology.orchestra.run.smithi129.stdout: 103 16 7710 7694 298.771 304 0.215575 0.214004 2024-09-15T20:27:37.890 INFO:teuthology.orchestra.run.smithi129.stdout: 104 16 7785 7769 298.783 300 0.181408 0.214029 2024-09-15T20:27:38.890 INFO:teuthology.orchestra.run.smithi129.stdout: 105 16 7857 7841 298.68 288 0.219413 0.214035 2024-09-15T20:27:39.890 INFO:teuthology.orchestra.run.smithi129.stdout: 106 16 7932 7916 298.692 300 0.22059 0.21403 2024-09-15T20:27:40.891 INFO:teuthology.orchestra.run.smithi129.stdout: 107 16 8008 7992 298.742 304 0.205626 0.214033 2024-09-15T20:27:41.891 INFO:teuthology.orchestra.run.smithi129.stdout: 108 16 8083 8067 298.753 300 0.189421 0.214045 2024-09-15T20:27:42.891 INFO:teuthology.orchestra.run.smithi129.stdout: 109 16 8155 8139 298.654 288 0.206111 0.214068 2024-09-15T20:27:43.891 INFO:teuthology.orchestra.run.smithi129.stdout: 110 16 8232 8216 298.739 308 0.193729 0.214035 2024-09-15T20:27:44.891 INFO:teuthology.orchestra.run.smithi129.stdout: 111 16 8304 8288 298.642 288 0.23267 0.214052 2024-09-15T20:27:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: 112 16 8379 8363 298.654 300 0.217925 0.2141 2024-09-15T20:27:46.891 INFO:teuthology.orchestra.run.smithi129.stdout: 113 16 8454 8438 298.666 300 0.1982 0.214067 2024-09-15T20:27:47.891 INFO:teuthology.orchestra.run.smithi129.stdout: 114 16 8529 8513 298.677 300 0.218587 0.214066 2024-09-15T20:27:48.891 INFO:teuthology.orchestra.run.smithi129.stdout: 115 16 8604 8588 298.688 300 0.21446 0.214071 2024-09-15T20:27:49.891 INFO:teuthology.orchestra.run.smithi129.stdout: 116 16 8678 8662 298.665 296 0.230444 0.214066 2024-09-15T20:27:50.891 INFO:teuthology.orchestra.run.smithi129.stdout: 117 16 8753 8737 298.676 300 0.206599 0.214079 2024-09-15T20:27:51.891 INFO:teuthology.orchestra.run.smithi129.stdout: 118 16 8827 8811 298.653 296 0.219725 0.214085 2024-09-15T20:27:52.892 INFO:teuthology.orchestra.run.smithi129.stdout: 119 16 8902 8886 298.664 300 0.216414 0.214094 2024-09-15T20:27:53.892 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:27:53.890802+0000 min lat: 0.0616332 max lat: 0.290218 avg lat: 0.2141 2024-09-15T20:27:53.892 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:27:53.892 INFO:teuthology.orchestra.run.smithi129.stdout: 120 16 8977 8961 298.675 300 0.186149 0.2141 2024-09-15T20:27:54.892 INFO:teuthology.orchestra.run.smithi129.stdout:Total time run: 120.195 2024-09-15T20:27:54.892 INFO:teuthology.orchestra.run.smithi129.stdout:Total writes made: 8977 2024-09-15T20:27:54.892 INFO:teuthology.orchestra.run.smithi129.stdout:Write size: 4194304 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Object size: 4194304 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Bandwidth (MB/sec): 298.748 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Stddev Bandwidth: 6.31205 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Max bandwidth (MB/sec): 316 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Min bandwidth (MB/sec): 284 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Average IOPS: 74 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Stddev IOPS: 1.57801 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Max IOPS: 79 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Min IOPS: 71 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Average Latency(s): 0.214083 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Stddev Latency(s): 0.0137193 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Max latency(s): 0.290218 2024-09-15T20:27:54.893 INFO:teuthology.orchestra.run.smithi129.stdout:Min latency(s): 0.0616332 2024-09-15T20:27:54.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd df name osd.1 --format=json 2024-09-15T20:27:55.195 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:27:55.195 INFO:teuthology.orchestra.run.smithi129.stdout:{"nodes":[{"id":1,"device_class":"ssd","name":"osd.1","type":"osd","type_id":0,"crush_weight":0.087890625,"depth":2,"pool_weights":{},"reweight":1,"kb":94371840,"kb_used":1126272,"kb_used_data":1095088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93245568,"utilization":1.1934407552083333,"var":1,"pgs":2,"status":"up"}],"stray":[],"summary":{"total_kb":94371840,"total_kb_used":1126272,"total_kb_used_data":1095088,"total_kb_used_omap":1,"total_kb_used_meta":31166,"total_kb_avail":93245568,"average_utilization":1.1934407552083333,"min_var":1,"max_var":1,"dev":0}} 2024-09-15T20:27:55.208 DEBUG:tasks.backfill_toofull:target osd df: {'id': 1, 'device_class': 'ssd', 'name': 'osd.1', 'type': 'osd', 'type_id': 0, 'crush_weight': 0.087890625, 'depth': 2, 'pool_weights': {}, 'reweight': 1, 'kb': 94371840, 'kb_used': 1126272, 'kb_used_data': 1095088, 'kb_used_omap': 1, 'kb_used_meta': 31166, 'kb_avail': 93245568, 'utilization': 1.1934407552083333, 'var': 1, 'pgs': 2, 'status': 'up'} 2024-09-15T20:27:55.208 DEBUG:tasks.backfill_toofull:pause recovery 2024-09-15T20:27:55.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noout 2024-09-15T20:27:55.817 INFO:teuthology.orchestra.run.smithi129.stderr:noout is set 2024-09-15T20:27:55.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nobackfill 2024-09-15T20:27:56.821 INFO:teuthology.orchestra.run.smithi129.stderr:nobackfill is set 2024-09-15T20:27:56.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set norecover 2024-09-15T20:27:57.825 INFO:teuthology.orchestra.run.smithi129.stderr:norecover is set 2024-09-15T20:27:57.840 DEBUG:tasks.backfill_toofull:stop tartget osd 1 2024-09-15T20:27:57.840 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T20:27:57.840 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:27:57.879 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T20:27:57.880 INFO:tasks.backfill_toofull.ceph_manager:waiting till active 2024-09-15T20:27:57.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:27:58.205 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:27:58.205 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:27:58.217 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-15T20:27:57.821363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37652595224,"num_objects":8980,"num_object_clones":0,"num_object_copies":26938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8980,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":9015,"num_write_kb":36770115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2470840,"kb_used_data":2328044,"kb_used_omap":6,"kb_used_meta":142713,"kb_avail":375016520,"statfs":{"total":386547056640,"available":384016916480,"internally_reserved":0,"allocated":2383917056,"data_stored":38132733256,"data_compressed":55814407,"data_compressed_allocated":2383282176,"data_compressed_original":38132514816,"omap_allocated":6378,"internal_metadata":146138902},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":977272856,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":234,"num_write_kb":954369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.241951"},"pg_stats":[{"pgid":"2.0","version":"18'8978","reported_seq":8998,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:27:55.950808+0000","last_change":"2024-09-15T20:25:47.649115+0000","last_active":"2024-09-15T20:27:55.950808+0000","last_peered":"2024-09-15T20:27:55.950808+0000","last_clean":"2024-09-15T20:27:55.950808+0000","last_became_active":"2024-09-15T20:25:47.648765+0000","last_became_peered":"2024-09-15T20:25:47.648765+0000","last_unstale":"2024-09-15T20:27:55.950808+0000","last_undegraded":"2024-09-15T20:27:55.950808+0000","last_fullsized":"2024-09-15T20:27:55.950808+0000","mapping_epoch":15,"log_start":"18'8900","ondisk_log_start":"18'8900","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":3000,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:24.230950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37652267032,"num_objects":8978,"num_object_clones":0,"num_object_copies":26934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":36769793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T20:25:51.284446+0000","last_change":"2024-09-15T20:25:35.238908+0000","last_active":"2024-09-15T20:25:51.284446+0000","last_peered":"2024-09-15T20:25:51.284446+0000","last_clean":"2024-09-15T20:25:51.284446+0000","last_became_active":"2024-09-15T20:25:35.237763+0000","last_became_peered":"2024-09-15T20:25:35.237763+0000","last_unstale":"2024-09-15T20:25:51.284446+0000","last_undegraded":"2024-09-15T20:25:51.284446+0000","last_fullsized":"2024-09-15T20:25:51.284446+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:48.064107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":37652267032,"num_objects":8978,"num_object_clones":0,"num_object_copies":26934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":36769793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2385821696,"data_stored":38173085696,"data_compressed":55872986,"data_compressed_allocated":2385817600,"data_compressed_original":38173081600,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1202604,"kb_used_data":1149212,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93169236,"statfs":{"total":96636764160,"available":95405297664,"internally_reserved":0,"allocated":1176793088,"data_stored":18826190930,"data_compressed":27556071,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67008,"kb_used_data":35824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304832,"statfs":{"total":96636764160,"available":96568147968,"internally_reserved":0,"allocated":36683776,"data_stored":584503378,"data_compressed":1118436,"data_compressed_allocated":36528128,"data_compressed_original":584450048,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705697,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174016,"kb_used_data":1142832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93197824,"statfs":{"total":96636764160,"available":95434571776,"internally_reserved":0,"allocated":1170259968,"data_stored":18721657426,"data_compressed":27138990,"data_compressed_allocated":1170100224,"data_compressed_original":18721603584,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705696,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":381522,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1172570112,"data_stored":18761121792,"data_compressed":27195840,"data_compressed_allocated":1172570112,"data_compressed_original":18761121792,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36614144,"data_stored":585826304,"data_compressed":1121075,"data_compressed_allocated":36614144,"data_compressed_original":585826304,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1176637440,"data_stored":18826137600,"data_compressed":27556071,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:27:58.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:27:58.568 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:27:58.568 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":155,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":4,"osd_up_since":1726431934,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":8980,"data_bytes":37652595224,"bytes_used":2530140160,"bytes_avail":384016916480,"bytes_total":386547056640,"write_bytes_sec":320295527,"read_op_per_sec":0,"write_op_per_sec":76},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-15T20:26:37.568593+0000","services":{}},"progress_events":{}} 2024-09-15T20:27:58.581 INFO:tasks.backfill_toofull.ceph_manager:{'fsid': '54dc298d-7b27-4c9d-9204-d7c9a8f9d562', 'health': {'status': 'HEALTH_WARN', 'checks': {'OSDMAP_FLAGS': {'severity': 'HEALTH_WARN', 'summary': {'message': 'noout,nobackfill,norecover flag(s) set', 'count': 26}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 155, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 21, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1726431934, 'num_in_osds': 4, 'osd_in_since': 1726431924, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 8980, 'data_bytes': 37652595224, 'bytes_used': 2530140160, 'bytes_avail': 384016916480, 'bytes_total': 386547056640, 'write_bytes_sec': 320295527, 'read_op_per_sec': 0, 'write_op_per_sec': 76}, 'fsmap': {'epoch': 1, 'btime': '2024-09-15T20:25:23:126945+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 4, 'modified': '2024-09-15T20:26:37.568593+0000', 'services': {}}, 'progress_events': {}} 2024-09-15T20:27:58.581 INFO:tasks.backfill_toofull.ceph_manager:active! 2024-09-15T20:27:58.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:27:58.865 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:27:58.866 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:27:58.878 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-15T20:27:58.825904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37652595224,"num_objects":8980,"num_object_clones":0,"num_object_copies":26938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8980,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":9015,"num_write_kb":36770115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2471116,"kb_used_data":2328316,"kb_used_omap":6,"kb_used_meta":142713,"kb_avail":375016244,"statfs":{"total":386547056640,"available":384016633856,"internally_reserved":0,"allocated":2384195584,"data_stored":38136619948,"data_compressed":55821820,"data_compressed_allocated":2383523840,"data_compressed_original":38136381440,"omap_allocated":6378,"internal_metadata":146138902},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":977272856,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":234,"num_write_kb":954369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.246195"},"pg_stats":[{"pgid":"2.0","version":"18'8978","reported_seq":8998,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:27:55.950808+0000","last_change":"2024-09-15T20:25:47.649115+0000","last_active":"2024-09-15T20:27:55.950808+0000","last_peered":"2024-09-15T20:27:55.950808+0000","last_clean":"2024-09-15T20:27:55.950808+0000","last_became_active":"2024-09-15T20:25:47.648765+0000","last_became_peered":"2024-09-15T20:25:47.648765+0000","last_unstale":"2024-09-15T20:27:55.950808+0000","last_undegraded":"2024-09-15T20:27:55.950808+0000","last_fullsized":"2024-09-15T20:27:55.950808+0000","mapping_epoch":15,"log_start":"18'8900","ondisk_log_start":"18'8900","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":3000,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:24.230950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37652267032,"num_objects":8978,"num_object_clones":0,"num_object_copies":26934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":36769793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"stale+active+clean","last_fresh":"2024-09-15T20:25:51.284446+0000","last_change":"2024-09-15T20:25:35.238908+0000","last_active":"2024-09-15T20:25:51.284446+0000","last_peered":"2024-09-15T20:25:51.284446+0000","last_clean":"2024-09-15T20:25:51.284446+0000","last_became_active":"2024-09-15T20:25:35.237763+0000","last_became_peered":"2024-09-15T20:25:35.237763+0000","last_unstale":"2024-09-15T20:27:58.825920+0000","last_undegraded":"2024-09-15T20:25:51.284446+0000","last_fullsized":"2024-09-15T20:25:51.284446+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:48.064107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":37652267032,"num_objects":8978,"num_object_clones":0,"num_object_copies":26934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":36769793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2385981440,"data_stored":38175580160,"data_compressed":55877760,"data_compressed_allocated":2385973248,"data_compressed_original":38175571968,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1202604,"kb_used_data":1149212,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93169236,"statfs":{"total":96636764160,"available":95405297664,"internally_reserved":0,"allocated":1176793088,"data_stored":18826190930,"data_compressed":27556071,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,1,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67272,"kb_used_data":36080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304568,"statfs":{"total":96636764160,"available":96567877632,"internally_reserved":0,"allocated":36945920,"data_stored":588382084,"data_compressed":1125849,"data_compressed_allocated":36769792,"data_compressed_original":588316672,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705697,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1174016,"kb_used_data":1142832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93197824,"statfs":{"total":96636764160,"available":95434571776,"internally_reserved":0,"allocated":1170259968,"data_stored":18721657426,"data_compressed":27138990,"data_compressed_allocated":1170100224,"data_compressed_original":18721603584,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705697,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":196608,"data_stored":389508,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1172570112,"data_stored":18761121792,"data_compressed":27195840,"data_compressed_allocated":1172570112,"data_compressed_original":18761121792,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36773888,"data_stored":588320768,"data_compressed":1125849,"data_compressed_allocated":36769792,"data_compressed_original":588316672,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1176637440,"data_stored":18826137600,"data_compressed":27556071,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:27:58.878 DEBUG:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "18'8978", 'reported_seq': 8998, 'reported_epoch': 19, 'state': 'active+clean', 'last_fresh': '2024-09-15T20:27:55.950808+0000', 'last_change': '2024-09-15T20:25:47.649115+0000', 'last_active': '2024-09-15T20:27:55.950808+0000', 'last_peered': '2024-09-15T20:27:55.950808+0000', 'last_clean': '2024-09-15T20:27:55.950808+0000', 'last_became_active': '2024-09-15T20:25:47.648765+0000', 'last_became_peered': '2024-09-15T20:25:47.648765+0000', 'last_unstale': '2024-09-15T20:27:55.950808+0000', 'last_undegraded': '2024-09-15T20:27:55.950808+0000', 'last_fullsized': '2024-09-15T20:27:55.950808+0000', 'mapping_epoch': 15, 'log_start': "18'8900", 'ondisk_log_start': "18'8900", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 78, 'log_dups_size': 3000, 'ondisk_log_size': 78, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T04:16:24.230950+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37652267032, 'num_objects': 8978, 'num_object_clones': 0, 'num_object_copies': 26934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8978, 'num_write_kb': 36769793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:27:58.879 DEBUG:tasks.backfill_toofull:re-write data 2024-09-15T20:27:58.879 INFO:tasks.util.rados:rados -p unique_pool_0 cleanup 2024-09-15T20:27:58.879 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 cleanup 2024-09-15T20:28:07.870 INFO:teuthology.orchestra.run.smithi129.stdout:Removed 8977 objects 2024-09-15T20:28:17.876 INFO:tasks.util.rados:rados -p unique_pool_0 bench 60 write --no-cleanup 2024-09-15T20:28:17.876 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 60 write --no-cleanup 2024-09-15T20:28:17.933 INFO:teuthology.orchestra.run.smithi129.stdout:hints = 1 2024-09-15T20:28:17.935 INFO:teuthology.orchestra.run.smithi129.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects 2024-09-15T20:28:17.935 INFO:teuthology.orchestra.run.smithi129.stdout:Object prefix: benchmark_data_smithi129_39215 2024-09-15T20:28:17.964 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:28:17.964 INFO:teuthology.orchestra.run.smithi129.stdout: 0 0 0 0 0 0 - 0 2024-09-15T20:28:18.964 INFO:teuthology.orchestra.run.smithi129.stdout: 1 16 94 78 311.973 312 0.193496 0.185078 2024-09-15T20:28:19.964 INFO:teuthology.orchestra.run.smithi129.stdout: 2 16 176 160 319.973 328 0.197497 0.190472 2024-09-15T20:28:20.964 INFO:teuthology.orchestra.run.smithi129.stdout: 3 16 256 240 319.973 320 0.207179 0.192612 2024-09-15T20:28:21.965 INFO:teuthology.orchestra.run.smithi129.stdout: 4 16 341 325 324.973 340 0.183189 0.192765 2024-09-15T20:28:22.965 INFO:teuthology.orchestra.run.smithi129.stdout: 5 16 422 406 324.772 324 0.193932 0.193129 2024-09-15T20:28:23.965 INFO:teuthology.orchestra.run.smithi129.stdout: 6 16 505 489 325.972 332 0.186442 0.193333 2024-09-15T20:28:24.965 INFO:teuthology.orchestra.run.smithi129.stdout: 7 16 585 569 325.116 320 0.190094 0.19394 2024-09-15T20:28:25.965 INFO:teuthology.orchestra.run.smithi129.stdout: 8 16 667 651 325.473 328 0.183363 0.194196 2024-09-15T20:28:26.965 INFO:teuthology.orchestra.run.smithi129.stdout: 9 16 749 733 325.751 328 0.197386 0.194376 2024-09-15T20:28:27.965 INFO:teuthology.orchestra.run.smithi129.stdout: 10 16 831 815 325.973 328 0.192826 0.194653 2024-09-15T20:28:28.965 INFO:teuthology.orchestra.run.smithi129.stdout: 11 16 914 898 326.518 332 0.171921 0.1946 2024-09-15T20:28:29.965 INFO:teuthology.orchestra.run.smithi129.stdout: 12 16 993 977 325.64 316 0.198811 0.194864 2024-09-15T20:28:30.965 INFO:teuthology.orchestra.run.smithi129.stdout: 13 16 1074 1058 325.512 324 0.204797 0.195068 2024-09-15T20:28:31.968 INFO:teuthology.orchestra.run.smithi129.stdout: 14 16 1157 1141 325.974 332 0.183795 0.195112 2024-09-15T20:28:32.965 INFO:teuthology.orchestra.run.smithi129.stdout: 15 16 1239 1223 326.107 328 0.189622 0.195056 2024-09-15T20:28:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: 16 16 1320 1304 325.973 324 0.194861 0.195076 2024-09-15T20:28:34.966 INFO:teuthology.orchestra.run.smithi129.stdout: 17 16 1402 1386 326.091 328 0.195726 0.195088 2024-09-15T20:28:35.966 INFO:teuthology.orchestra.run.smithi129.stdout: 18 16 1483 1467 325.974 324 0.195548 0.195215 2024-09-15T20:28:36.966 INFO:teuthology.orchestra.run.smithi129.stdout: 19 16 1566 1550 326.289 332 0.184241 0.195213 2024-09-15T20:28:37.966 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:28:37.965053+0000 min lat: 0.0702217 max lat: 0.243785 avg lat: 0.195185 2024-09-15T20:28:37.966 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:28:37.966 INFO:teuthology.orchestra.run.smithi129.stdout: 20 16 1647 1631 326.173 324 0.190305 0.195185 2024-09-15T20:28:38.966 INFO:teuthology.orchestra.run.smithi129.stdout: 21 16 1729 1713 326.259 328 0.192617 0.195285 2024-09-15T20:28:39.966 INFO:teuthology.orchestra.run.smithi129.stdout: 22 16 1810 1794 326.155 324 0.182184 0.195316 2024-09-15T20:28:40.966 INFO:teuthology.orchestra.run.smithi129.stdout: 23 16 1893 1877 326.408 332 0.187213 0.195309 2024-09-15T20:28:41.966 INFO:teuthology.orchestra.run.smithi129.stdout: 24 16 1975 1959 326.473 328 0.203113 0.195253 2024-09-15T20:28:42.966 INFO:teuthology.orchestra.run.smithi129.stdout: 25 16 2057 2041 326.533 328 0.173715 0.195283 2024-09-15T20:28:43.966 INFO:teuthology.orchestra.run.smithi129.stdout: 26 16 2135 2119 325.973 312 0.206295 0.195532 2024-09-15T20:28:44.966 INFO:teuthology.orchestra.run.smithi129.stdout: 27 16 2216 2200 325.899 324 0.173876 0.195604 2024-09-15T20:28:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: 28 16 2298 2282 325.973 328 0.186709 0.195695 2024-09-15T20:28:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: 29 16 2379 2363 325.904 324 0.204565 0.195748 2024-09-15T20:28:47.967 INFO:teuthology.orchestra.run.smithi129.stdout: 30 16 2460 2444 325.84 324 0.19682 0.19575 2024-09-15T20:28:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: 31 16 2541 2525 325.78 324 0.196321 0.195792 2024-09-15T20:28:49.967 INFO:teuthology.orchestra.run.smithi129.stdout: 32 16 2624 2608 325.973 332 0.182631 0.19573 2024-09-15T20:28:50.967 INFO:teuthology.orchestra.run.smithi129.stdout: 33 16 2706 2690 326.034 328 0.192534 0.195739 2024-09-15T20:28:51.969 INFO:teuthology.orchestra.run.smithi129.stdout: 34 16 2788 2772 326.091 328 0.172356 0.195688 2024-09-15T20:28:52.967 INFO:teuthology.orchestra.run.smithi129.stdout: 35 16 2870 2854 326.145 328 0.182624 0.19569 2024-09-15T20:28:53.967 INFO:teuthology.orchestra.run.smithi129.stdout: 36 16 2952 2936 326.196 328 0.185979 0.195725 2024-09-15T20:28:54.967 INFO:teuthology.orchestra.run.smithi129.stdout: 37 16 3033 3017 326.135 324 0.192632 0.195731 2024-09-15T20:28:55.967 INFO:teuthology.orchestra.run.smithi129.stdout: 38 16 3112 3096 325.868 316 0.19841 0.195795 2024-09-15T20:28:56.967 INFO:teuthology.orchestra.run.smithi129.stdout: 39 16 3195 3179 326.025 332 0.176597 0.195789 2024-09-15T20:28:57.968 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:28:57.966696+0000 min lat: 0.0702217 max lat: 0.246843 avg lat: 0.195835 2024-09-15T20:28:57.968 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:28:57.968 INFO:teuthology.orchestra.run.smithi129.stdout: 40 16 3276 3260 325.973 324 0.195833 0.195835 2024-09-15T20:28:58.968 INFO:teuthology.orchestra.run.smithi129.stdout: 41 16 3358 3342 326.022 328 0.198011 0.195832 2024-09-15T20:28:59.968 INFO:teuthology.orchestra.run.smithi129.stdout: 42 16 3438 3422 325.878 320 0.213427 0.195883 2024-09-15T20:29:00.968 INFO:teuthology.orchestra.run.smithi129.stdout: 43 16 3520 3504 325.927 328 0.197429 0.195934 2024-09-15T20:29:01.968 INFO:teuthology.orchestra.run.smithi129.stdout: 44 16 3601 3585 325.882 324 0.192881 0.195899 2024-09-15T20:29:02.968 INFO:teuthology.orchestra.run.smithi129.stdout: 45 16 3683 3667 325.929 328 0.204052 0.195913 2024-09-15T20:29:03.968 INFO:teuthology.orchestra.run.smithi129.stdout: 46 16 3763 3747 325.799 320 0.217653 0.195935 2024-09-15T20:29:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: 47 16 3845 3829 325.846 328 0.194268 0.196 2024-09-15T20:29:05.968 INFO:teuthology.orchestra.run.smithi129.stdout: 48 16 3925 3909 325.723 320 0.195451 0.195999 2024-09-15T20:29:06.968 INFO:teuthology.orchestra.run.smithi129.stdout: 49 16 4009 3993 325.932 336 0.172241 0.196011 2024-09-15T20:29:07.969 INFO:teuthology.orchestra.run.smithi129.stdout: 50 16 4090 4074 325.893 324 0.194161 0.19602 2024-09-15T20:29:08.968 INFO:teuthology.orchestra.run.smithi129.stdout: 51 16 4172 4156 325.934 328 0.193239 0.195984 2024-09-15T20:29:09.969 INFO:teuthology.orchestra.run.smithi129.stdout: 52 16 4253 4237 325.896 324 0.194383 0.196001 2024-09-15T20:29:10.969 INFO:teuthology.orchestra.run.smithi129.stdout: 53 16 4334 4318 325.86 324 0.197565 0.196006 2024-09-15T20:29:11.969 INFO:teuthology.orchestra.run.smithi129.stdout: 54 16 4416 4400 325.899 328 0.196841 0.196026 2024-09-15T20:29:12.969 INFO:teuthology.orchestra.run.smithi129.stdout: 55 16 4497 4481 325.864 324 0.185475 0.196067 2024-09-15T20:29:13.969 INFO:teuthology.orchestra.run.smithi129.stdout: 56 16 4577 4561 325.759 320 0.188598 0.196083 2024-09-15T20:29:14.969 INFO:teuthology.orchestra.run.smithi129.stdout: 57 16 4659 4643 325.798 328 0.18354 0.196084 2024-09-15T20:29:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: 58 16 4741 4725 325.835 328 0.198827 0.196103 2024-09-15T20:29:16.969 INFO:teuthology.orchestra.run.smithi129.stdout: 59 16 4821 4805 325.736 320 0.198843 0.196157 2024-09-15T20:29:17.969 INFO:teuthology.orchestra.run.smithi129.stdout:2024-09-15T20:29:17.968375+0000 min lat: 0.0702217 max lat: 0.246843 avg lat: 0.196173 2024-09-15T20:29:17.969 INFO:teuthology.orchestra.run.smithi129.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:29:17.969 INFO:teuthology.orchestra.run.smithi129.stdout: 60 16 4903 4887 325.773 328 0.173929 0.196173 2024-09-15T20:29:18.969 INFO:teuthology.orchestra.run.smithi129.stdout:Total time run: 60.1741 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Total writes made: 4903 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Write size: 4194304 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Object size: 4194304 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Bandwidth (MB/sec): 325.921 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Stddev Bandwidth: 5.12488 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Max bandwidth (MB/sec): 340 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Min bandwidth (MB/sec): 312 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Average IOPS: 81 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Stddev IOPS: 1.28122 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Max IOPS: 85 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Min IOPS: 78 2024-09-15T20:29:18.970 INFO:teuthology.orchestra.run.smithi129.stdout:Average Latency(s): 0.196136 2024-09-15T20:29:18.971 INFO:teuthology.orchestra.run.smithi129.stdout:Stddev Latency(s): 0.0119937 2024-09-15T20:29:18.971 INFO:teuthology.orchestra.run.smithi129.stdout:Max latency(s): 0.246843 2024-09-15T20:29:18.971 INFO:teuthology.orchestra.run.smithi129.stdout:Min latency(s): 0.0702217 2024-09-15T20:29:18.978 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd df name osd.3 --format=json 2024-09-15T20:29:19.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:19.274 INFO:teuthology.orchestra.run.smithi129.stdout:{"nodes":[{"id":3,"device_class":"ssd","name":"osd.3","type":"osd","type_id":0,"crush_weight":0.087890625,"depth":2,"pool_weights":{},"reweight":1,"kb":94371840,"kb_used":701008,"kb_used_data":603200,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93670832,"utilization":0.74281480577256942,"var":1,"pgs":1,"status":"up"}],"stray":[],"summary":{"total_kb":94371840,"total_kb_used":701008,"total_kb_used_data":603200,"total_kb_used_omap":1,"total_kb_used_meta":97790,"total_kb_avail":93670832,"average_utilization":0.74281480577256942,"min_var":1,"max_var":1,"dev":0}} 2024-09-15T20:29:19.287 DEBUG:tasks.backfill_toofull:primary osd df: {'id': 3, 'device_class': 'ssd', 'name': 'osd.3', 'type': 'osd', 'type_id': 0, 'crush_weight': 0.087890625, 'depth': 2, 'pool_weights': {}, 'reweight': 1, 'kb': 94371840, 'kb_used': 701008, 'kb_used_data': 603200, 'kb_used_omap': 1, 'kb_used_meta': 97790, 'kb_avail': 93670832, 'utilization': 0.7428148057725694, 'var': 1, 'pgs': 1, 'status': 'up'} 2024-09-15T20:29:19.288 INFO:tasks.backfill_toofull:test backfill reservation rejected with toofull 2024-09-15T20:29:19.288 DEBUG:tasks.backfill_toofull:update nearfull ratio to 0.0060167999267578135 and backfillfull ratio to 0.006685333251953126 2024-09-15T20:29:19.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-nearfull-ratio 0.007 2024-09-15T20:29:20.272 INFO:teuthology.orchestra.run.smithi129.stderr:osd set-nearfull-ratio 0.007 2024-09-15T20:29:20.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-backfillfull-ratio 0.008 2024-09-15T20:29:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:osd set-backfillfull-ratio 0.008 2024-09-15T20:29:21.294 DEBUG:tasks.backfill_toofull:start tartget osd 1 2024-09-15T20:29:21.294 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T20:29:21.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T20:29:21.297 INFO:tasks.ceph.osd.1:Started 2024-09-15T20:29:21.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:no valid command found; 10 closest matches: 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:0 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:1 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:2 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:abort 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:assert 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:config diff 2024-09-15T20:29:21.400 INFO:teuthology.orchestra.run.smithi129.stderr:config diff get 2024-09-15T20:29:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:config get 2024-09-15T20:29:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:config help [] 2024-09-15T20:29:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:config set ... 2024-09-15T20:29:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:admin_socket: invalid command 2024-09-15T20:29:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T20:29:21.403 INFO:tasks.backfill_toofull.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-15T20:29:22.506 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:29:22.504+0000 7f0dc3db0640 -1 Falling back to public interface 2024-09-15T20:29:26.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T20:29:26.506 INFO:teuthology.orchestra.run.smithi129.stderr:no valid command found; 10 closest matches: 2024-09-15T20:29:26.508 INFO:teuthology.orchestra.run.smithi129.stderr:0 2024-09-15T20:29:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:1 2024-09-15T20:29:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:2 2024-09-15T20:29:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:abort 2024-09-15T20:29:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:assert 2024-09-15T20:29:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:bluefs debug_inject_read_zeros 2024-09-15T20:29:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:bluefs files list 2024-09-15T20:29:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:bluefs stats 2024-09-15T20:29:26.510 INFO:teuthology.orchestra.run.smithi129.stderr:bluestore allocator dump block 2024-09-15T20:29:26.510 INFO:teuthology.orchestra.run.smithi129.stderr:bluestore allocator fragmentation block 2024-09-15T20:29:26.510 INFO:teuthology.orchestra.run.smithi129.stderr:admin_socket: invalid command 2024-09-15T20:29:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T20:29:26.512 INFO:tasks.backfill_toofull.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-15T20:29:28.928 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:29:28.925+0000 7f0dc3db0640 -1 osd.1 21 log_to_monitors true 2024-09-15T20:29:31.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T20:29:31.619 INFO:teuthology.orchestra.run.smithi129.stderr:no valid command found; 10 closest matches: 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:0 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:1 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:2 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:abort 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:assert 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:bluefs debug_inject_read_zeros 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:bluefs files list 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:bluefs stats 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:bluestore allocator dump block 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:bluestore allocator fragmentation block 2024-09-15T20:29:31.620 INFO:teuthology.orchestra.run.smithi129.stderr:admin_socket: invalid command 2024-09-15T20:29:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T20:29:31.623 INFO:tasks.backfill_toofull.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-15T20:29:31.849 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:29:31.846+0000 7f0db7b5e640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:29:36.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T20:29:36.739 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-15T20:29:36.740 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-15T20:29:36.740 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-15T20:29:36.740 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-15T20:29:36.748 INFO:tasks.backfill_toofull.ceph_manager:waiting for peering to complete 2024-09-15T20:29:36.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:29:37.028 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:37.029 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:29:37.041 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-15T20:29:35.597134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2019812,"kb_used_data":1797252,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467548,"statfs":{"total":386547056640,"available":384478769152,"internally_reserved":0,"allocated":1840386048,"data_stored":29430848602,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001036"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T20:29:29.290000+0000","last_change":"2024-09-15T20:28:18.033676+0000","last_active":"2024-09-15T20:29:29.290000+0000","last_peered":"2024-09-15T20:29:29.290000+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:27:59.835893+0000","last_became_peered":"2024-09-15T20:27:59.835893+0000","last_unstale":"2024-09-15T20:29:29.290000+0000","last_undegraded":"2024-09-15T20:28:18.033516+0000","last_fullsized":"2024-09-15T20:27:59.833329+0000","mapping_epoch":22,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:10.961065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-15T20:29:34.126685+0000","last_change":"2024-09-15T20:29:33.126879+0000","last_active":"2024-09-15T20:29:34.126685+0000","last_peered":"2024-09-15T20:29:34.126685+0000","last_clean":"2024-09-15T20:29:34.126685+0000","last_became_active":"2024-09-15T20:29:33.126528+0000","last_became_peered":"2024-09-15T20:29:33.126528+0000","last_unstale":"2024-09-15T20:29:34.126685+0000","last_undegraded":"2024-09-15T20:29:34.126685+0000","last_fullsized":"2024-09-15T20:29:34.126685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:08.279124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640309,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708984,"kb_used_data":627812,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662856,"statfs":{"total":96636764160,"available":95910764544,"internally_reserved":0,"allocated":642879488,"data_stored":10282431250,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95464,"kb_used_data":19856,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276376,"statfs":{"total":96636764160,"available":96539009024,"internally_reserved":0,"allocated":20332544,"data_stored":321427224,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084290,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188068,"kb_used_data":1149320,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183772,"statfs":{"total":96636764160,"available":95420182528,"internally_reserved":0,"allocated":1176903680,"data_stored":18826561816,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":428312,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:29:37.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:29:37.321 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:37.321 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:29:37.333 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-15T20:29:35.597134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2019812,"kb_used_data":1797252,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467548,"statfs":{"total":386547056640,"available":384478769152,"internally_reserved":0,"allocated":1840386048,"data_stored":29430848602,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001036"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22902,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T20:29:29.290000+0000","last_change":"2024-09-15T20:28:18.033676+0000","last_active":"2024-09-15T20:29:29.290000+0000","last_peered":"2024-09-15T20:29:29.290000+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:27:59.835893+0000","last_became_peered":"2024-09-15T20:27:59.835893+0000","last_unstale":"2024-09-15T20:29:29.290000+0000","last_undegraded":"2024-09-15T20:28:18.033516+0000","last_fullsized":"2024-09-15T20:27:59.833329+0000","mapping_epoch":22,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:10.961065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-15T20:29:34.126685+0000","last_change":"2024-09-15T20:29:33.126879+0000","last_active":"2024-09-15T20:29:34.126685+0000","last_peered":"2024-09-15T20:29:34.126685+0000","last_clean":"2024-09-15T20:29:34.126685+0000","last_became_active":"2024-09-15T20:29:33.126528+0000","last_became_peered":"2024-09-15T20:29:33.126528+0000","last_unstale":"2024-09-15T20:29:34.126685+0000","last_undegraded":"2024-09-15T20:29:34.126685+0000","last_fullsized":"2024-09-15T20:29:34.126685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:08.279124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640309,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708984,"kb_used_data":627812,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662856,"statfs":{"total":96636764160,"available":95910764544,"internally_reserved":0,"allocated":642879488,"data_stored":10282431250,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95464,"kb_used_data":19856,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276376,"statfs":{"total":96636764160,"available":96539009024,"internally_reserved":0,"allocated":20332544,"data_stored":321427224,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084290,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188068,"kb_used_data":1149320,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183772,"statfs":{"total":96636764160,"available":95420182528,"internally_reserved":0,"allocated":1176903680,"data_stored":18826561816,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":428312,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:29:37.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:29:37.685 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:37.685 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout,nobackfill,norecover flag(s) set","count":26},"muted":false},"OSD_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"1 backfillfull osd(s)","count":1},"muted":false},"OSD_NEARFULL":{"severity":"HEALTH_WARN","summary":{"message":"1 nearfull osd(s)","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4904/14716 objects degraded (33.324%), 1 pg degraded, 1 pg undersized","count":2},"muted":false},"POOL_BACKFILLFULL":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) backfillfull","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":254,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":30,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432172,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":2068287488,"bytes_avail":384478769152,"bytes_total":386547056640,"degraded_objects":4904,"degraded_total":14716,"degraded_ratio":0.33324272900244639},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-15T20:26:37.568593+0000","services":{}},"progress_events":{}} 2024-09-15T20:29:37.698 INFO:tasks.backfill_toofull.ceph_manager:{'fsid': '54dc298d-7b27-4c9d-9204-d7c9a8f9d562', 'health': {'status': 'HEALTH_WARN', 'checks': {'OSDMAP_FLAGS': {'severity': 'HEALTH_WARN', 'summary': {'message': 'noout,nobackfill,norecover flag(s) set', 'count': 26}, 'muted': False}, 'OSD_BACKFILLFULL': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 backfillfull osd(s)', 'count': 1}, 'muted': False}, 'OSD_NEARFULL': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 nearfull osd(s)', 'count': 1}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 4904/14716 objects degraded (33.324%), 1 pg degraded, 1 pg undersized', 'count': 2}, 'muted': False}, 'POOL_BACKFILLFULL': {'severity': 'HEALTH_WARN', 'summary': {'message': '2 pool(s) backfillfull', 'count': 2}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 254, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 30, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1726432172, 'num_in_osds': 4, 'osd_in_since': 1726431924, 'num_remapped_pgs': 1}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+undersized+degraded', 'count': 1}, {'state_name': 'active+clean', 'count': 1}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 4906, 'data_bytes': 20565000728, 'bytes_used': 2068287488, 'bytes_avail': 384478769152, 'bytes_total': 386547056640, 'degraded_objects': 4904, 'degraded_total': 14716, 'degraded_ratio': 0.3332427290024464}, 'fsmap': {'epoch': 1, 'btime': '2024-09-15T20:25:23:126945+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 4, 'modified': '2024-09-15T20:26:37.568593+0000', 'services': {}}, 'progress_events': {}} 2024-09-15T20:29:37.698 INFO:tasks.backfill_toofull.ceph_manager:active! 2024-09-15T20:29:37.698 INFO:tasks.backfill_toofull.ceph_manager:waiting for osd.1 to be up 2024-09-15T20:29:37.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:29:37.980 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:37.981 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":30,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:29:34.120632+0000","last_up_change":"2024-09-15T20:29:32.843829+0000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"noout,nobackfill,norecover,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801600,"flags_set":["nobackfill","noout","norecover","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.0080000003799796104,"nearfull_ratio":0.0070000002160668373,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:25:33.574537+0000","flags":4097,"flags_names":"hashpspool,backfillfull","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2024-09-15T20:25:45.636932+0000","flags":4097,"flags_names":"hashpspool,backfillfull","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_toofull","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6809","nonce":4274430252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6811","nonce":4274430252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6815","nonce":4274430252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6813","nonce":4274430252}]},"public_addr":"172.21.15.129:6809/4274430252","cluster_addr":"172.21.15.129:6811/4274430252","heartbeat_back_addr":"172.21.15.129:6815/4274430252","heartbeat_front_addr":"172.21.15.129:6813/4274430252","state":["exists","up"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":21,"up_from":28,"up_thru":28,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1328262137},{"type":"v1","addr":"172.21.15.129:6801","nonce":1328262137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":1328262137},{"type":"v1","addr":"172.21.15.129:6803","nonce":1328262137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1328262137},{"type":"v1","addr":"172.21.15.129:6807","nonce":1328262137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1328262137},{"type":"v1","addr":"172.21.15.129:6805","nonce":1328262137}]},"public_addr":"172.21.15.129:6801/1328262137","cluster_addr":"172.21.15.129:6803/1328262137","heartbeat_back_addr":"172.21.15.129:6807/1328262137","heartbeat_front_addr":"172.21.15.129:6805/1328262137","state":["backfillfull","exists","up"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6825","nonce":3344078257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6827","nonce":3344078257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6831","nonce":3344078257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6829","nonce":3344078257}]},"public_addr":"172.21.15.129:6825/3344078257","cluster_addr":"172.21.15.129:6827/3344078257","heartbeat_back_addr":"172.21.15.129:6831/3344078257","heartbeat_front_addr":"172.21.15.129:6829/3344078257","state":["exists","up"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6817","nonce":435893452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6819","nonce":435893452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6823","nonce":435893452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6821","nonce":435893452}]},"public_addr":"172.21.15.129:6817/435893452","cluster_addr":"172.21.15.129:6819/435893452","heartbeat_back_addr":"172.21.15.129:6823/435893452","heartbeat_front_addr":"172.21.15.129:6821/435893452","state":["exists","nearfull","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.577650+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:30.816456+0000","dead_epoch":22},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.038929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.251155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"2.0","osds":[3,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_toofull":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:29:37.993 INFO:tasks.backfill_toofull.ceph_manager:osd.1 is up 2024-09-15T20:29:37.993 DEBUG:tasks.backfill_toofull:waiting for pg 2.0 state is backfill_toofull 2024-09-15T20:29:42.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T20:29:42.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T20:29:42.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T20:29:42.996 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T20:29:43.149 INFO:teuthology.orchestra.run.smithi129.stdout:38654705718 2024-09-15T20:29:43.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:29:43.151 INFO:teuthology.orchestra.run.smithi129.stdout:120259084292 2024-09-15T20:29:43.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:29:43.168 INFO:teuthology.orchestra.run.smithi129.stdout:47244640312 2024-09-15T20:29:43.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:29:43.169 INFO:teuthology.orchestra.run.smithi129.stdout:38654705718 2024-09-15T20:29:43.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:29:43.459 INFO:teuthology.orchestra.run.smithi129.stdout:120259084291 2024-09-15T20:29:43.477 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084292 got 120259084291 for osd.1 2024-09-15T20:29:43.499 INFO:teuthology.orchestra.run.smithi129.stdout:38654705717 2024-09-15T20:29:43.510 INFO:teuthology.orchestra.run.smithi129.stdout:47244640311 2024-09-15T20:29:43.512 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705718 got 38654705717 for osd.2 2024-09-15T20:29:43.522 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640312 got 47244640311 for osd.3 2024-09-15T20:29:43.550 INFO:teuthology.orchestra.run.smithi129.stdout:38654705717 2024-09-15T20:29:43.563 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705718 got 38654705717 for osd.0 2024-09-15T20:29:44.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:29:44.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:29:44.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:29:44.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:29:44.827 INFO:teuthology.orchestra.run.smithi129.stdout:120259084292 2024-09-15T20:29:44.830 INFO:teuthology.orchestra.run.smithi129.stdout:47244640312 2024-09-15T20:29:44.831 INFO:teuthology.orchestra.run.smithi129.stdout:38654705719 2024-09-15T20:29:44.844 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084292 got 120259084292 for osd.1 2024-09-15T20:29:44.844 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:44.846 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705718 got 38654705719 for osd.2 2024-09-15T20:29:44.846 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:44.851 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640312 got 47244640312 for osd.3 2024-09-15T20:29:44.851 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:44.903 INFO:teuthology.orchestra.run.smithi129.stdout:38654705719 2024-09-15T20:29:44.916 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705718 got 38654705719 for osd.0 2024-09-15T20:29:44.916 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:44.916 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:29:45.195 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:45.195 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:29:45.208 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-15T20:29:43.598744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2019856,"kb_used_data":1797300,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467504,"statfs":{"total":386547056640,"available":384478724096,"internally_reserved":0,"allocated":1840435200,"data_stored":29430873960,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002040"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22914,"reported_epoch":30,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-15T20:29:34.150898+0000","last_change":"2024-09-15T20:29:34.150898+0000","last_active":"2024-09-15T20:29:34.150898+0000","last_peered":"2024-09-15T20:29:34.150898+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:29:34.149984+0000","last_became_peered":"2024-09-15T20:29:34.149984+0000","last_unstale":"2024-09-15T20:29:34.150898+0000","last_undegraded":"2024-09-15T20:29:34.131101+0000","last_fullsized":"2024-09-15T20:29:34.128409+0000","mapping_epoch":29,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-15T20:29:34.126685+0000","last_change":"2024-09-15T20:29:33.126879+0000","last_active":"2024-09-15T20:29:34.126685+0000","last_peered":"2024-09-15T20:29:34.126685+0000","last_clean":"2024-09-15T20:29:34.126685+0000","last_became_active":"2024-09-15T20:29:33.126528+0000","last_became_peered":"2024-09-15T20:29:33.126528+0000","last_unstale":"2024-09-15T20:29:34.126685+0000","last_undegraded":"2024-09-15T20:29:34.126685+0000","last_fullsized":"2024-09-15T20:29:34.126685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:08.279124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640312,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709008,"kb_used_data":627836,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662832,"statfs":{"total":96636764160,"available":95910739968,"internally_reserved":0,"allocated":642904064,"data_stored":10282444506,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95468,"kb_used_data":19864,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276372,"statfs":{"total":96636764160,"available":96539004928,"internally_reserved":0,"allocated":20340736,"data_stored":321431258,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188080,"kb_used_data":1149328,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183760,"statfs":{"total":96636764160,"available":95420170240,"internally_reserved":0,"allocated":1176911872,"data_stored":18826565850,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":432346,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:29:45.209 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'22860", 'reported_seq': 22914, 'reported_epoch': 30, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2024-09-15T20:29:34.150898+0000', 'last_change': '2024-09-15T20:29:34.150898+0000', 'last_active': '2024-09-15T20:29:34.150898+0000', 'last_peered': '2024-09-15T20:29:34.150898+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:29:34.149984+0000', 'last_became_peered': '2024-09-15T20:29:34.149984+0000', 'last_unstale': '2024-09-15T20:29:34.150898+0000', 'last_undegraded': '2024-09-15T20:29:34.131101+0000', 'last_fullsized': '2024-09-15T20:29:34.128409+0000', 'mapping_epoch': 29, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:29:45.209 INFO:tasks.backfill_toofull:test pg not enter backfill_toofull after restarting backfill 2024-09-15T20:29:45.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df detail --format=json 2024-09-15T20:29:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:45.561 INFO:teuthology.orchestra.run.smithi129.stdout:{"stats":{"total_bytes":386547056640,"total_avail_bytes":384478724096,"total_used_bytes":2068332544,"total_used_raw_bytes":2068332544,"total_used_raw_ratio":0.0053507909178733826,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4},"stats_by_class":{"ssd":{"total_bytes":386547056640,"total_avail_bytes":384478724096,"total_used_bytes":2068332544,"total_used_raw_bytes":2068332544,"total_used_raw_ratio":0.0053507909178733826}},"pools":[{"name":".mgr","id":1,"stats":{"stored":328192,"stored_data":328192,"stored_omap":0,"objects":2,"kb_used":48,"bytes_used":49152,"data_bytes_used":49152,"omap_bytes_used":0,"percent_used":1.3564658729592338e-07,"max_avail":181176664064,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":26,"rd_bytes":19456,"wr":37,"wr_bytes":329728,"compress_bytes_used":40960,"compress_under_bytes":655360,"stored_raw":656384,"avail_raw":362353323520}},{"name":"unique_pool_0","id":2,"stats":{"stored":19619866624,"stored_data":19619866624,"stored_omap":0,"objects":4904,"kb_used":1796260,"bytes_used":1839370240,"data_bytes_used":1839370240,"omap_bytes_used":0,"percent_used":0.0050505413673818111,"max_avail":241568890880,"quota_objects":0,"quota_bytes":0,"dirty":0,"rd":1,"rd_bytes":1024,"wr":22860,"wr_bytes":58216941568,"compress_bytes_used":1839362048,"compress_under_bytes":29429792768,"stored_raw":29429800960,"avail_raw":362353323520}}]} 2024-09-15T20:29:45.574 DEBUG:tasks.backfill_toofull:pool unique_pool_0 df: {'stored': 19619866624, 'stored_data': 19619866624, 'stored_omap': 0, 'objects': 4904, 'kb_used': 1796260, 'bytes_used': 1839370240, 'data_bytes_used': 1839370240, 'omap_bytes_used': 0, 'percent_used': 0.005050541367381811, 'max_avail': 241568890880, 'quota_objects': 0, 'quota_bytes': 0, 'dirty': 0, 'rd': 1, 'rd_bytes': 1024, 'wr': 22860, 'wr_bytes': 58216941568, 'compress_bytes_used': 1839362048, 'compress_under_bytes': 29429792768, 'stored_raw': 29429800960, 'avail_raw': 362353323520} 2024-09-15T20:29:45.574 DEBUG:tasks.backfill_toofull:compress_ratio: 16.0 2024-09-15T20:29:45.574 DEBUG:tasks.backfill_toofull:update nearfull ratio to 0.20283555772569445 and backfillfull ratio to 0.21472059461805554 2024-09-15T20:29:45.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-nearfull-ratio 0.204 2024-09-15T20:29:46.204 INFO:tasks.ceph.mon.c.smithi129.stderr:2024-09-15T20:29:46.202+0000 7fcd4b20a640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-15T20:29:46.205 INFO:teuthology.orchestra.run.smithi129.stderr:osd set-nearfull-ratio 0.204 2024-09-15T20:29:46.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-backfillfull-ratio 0.216 2024-09-15T20:29:47.216 INFO:teuthology.orchestra.run.smithi129.stderr:osd set-backfillfull-ratio 0.216 2024-09-15T20:29:47.231 DEBUG:tasks.backfill_toofull:waiting for pg 2.0 state is backfilling 2024-09-15T20:29:52.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T20:29:52.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T20:29:52.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T20:29:52.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T20:29:52.393 INFO:teuthology.orchestra.run.smithi129.stdout:47244640315 2024-09-15T20:29:52.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:29:52.395 INFO:teuthology.orchestra.run.smithi129.stdout:120259084295 2024-09-15T20:29:52.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:29:52.396 INFO:teuthology.orchestra.run.smithi129.stdout:38654705721 2024-09-15T20:29:52.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:29:52.397 INFO:teuthology.orchestra.run.smithi129.stdout:38654705721 2024-09-15T20:29:52.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:29:52.740 INFO:teuthology.orchestra.run.smithi129.stdout:47244640314 2024-09-15T20:29:52.747 INFO:teuthology.orchestra.run.smithi129.stdout:38654705720 2024-09-15T20:29:52.755 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640315 got 47244640314 for osd.3 2024-09-15T20:29:52.762 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705721 got 38654705720 for osd.2 2024-09-15T20:29:52.785 INFO:teuthology.orchestra.run.smithi129.stdout:38654705720 2024-09-15T20:29:52.786 INFO:teuthology.orchestra.run.smithi129.stdout:120259084294 2024-09-15T20:29:52.800 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705721 got 38654705720 for osd.0 2024-09-15T20:29:52.801 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084295 got 120259084294 for osd.1 2024-09-15T20:29:53.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:29:53.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:29:53.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:29:53.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:29:54.089 INFO:teuthology.orchestra.run.smithi129.stdout:47244640314 2024-09-15T20:29:54.106 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640315 got 47244640314 for osd.3 2024-09-15T20:29:54.125 INFO:teuthology.orchestra.run.smithi129.stdout:38654705720 2024-09-15T20:29:54.125 INFO:teuthology.orchestra.run.smithi129.stdout:38654705720 2024-09-15T20:29:54.139 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705721 got 38654705720 for osd.2 2024-09-15T20:29:54.139 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705721 got 38654705720 for osd.0 2024-09-15T20:29:54.164 INFO:teuthology.orchestra.run.smithi129.stdout:120259084294 2024-09-15T20:29:54.176 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084295 got 120259084294 for osd.1 2024-09-15T20:29:55.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:29:55.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:29:55.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:29:55.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:29:55.439 INFO:teuthology.orchestra.run.smithi129.stdout:47244640315 2024-09-15T20:29:55.453 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640315 got 47244640315 for osd.3 2024-09-15T20:29:55.453 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:55.467 INFO:teuthology.orchestra.run.smithi129.stdout:38654705722 2024-09-15T20:29:55.481 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705721 got 38654705722 for osd.2 2024-09-15T20:29:55.481 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:55.507 INFO:teuthology.orchestra.run.smithi129.stdout:38654705722 2024-09-15T20:29:55.524 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705721 got 38654705722 for osd.0 2024-09-15T20:29:55.524 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:55.548 INFO:teuthology.orchestra.run.smithi129.stdout:120259084295 2024-09-15T20:29:55.560 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084295 got 120259084295 for osd.1 2024-09-15T20:29:55.560 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:55.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:29:55.884 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:29:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:29:55.897 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-15T20:29:55.600988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2019984,"kb_used_data":1797428,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467376,"statfs":{"total":386547056640,"available":384478593024,"internally_reserved":0,"allocated":1840566272,"data_stored":29430944248,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001609"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22922,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-15T20:29:49.229908+0000","last_change":"2024-09-15T20:29:34.150898+0000","last_active":"2024-09-15T20:29:49.229908+0000","last_peered":"2024-09-15T20:29:49.229908+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:29:34.149984+0000","last_became_peered":"2024-09-15T20:29:34.149984+0000","last_unstale":"2024-09-15T20:29:49.229908+0000","last_undegraded":"2024-09-15T20:29:34.131101+0000","last_fullsized":"2024-09-15T20:29:34.128409+0000","mapping_epoch":29,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T20:29:49.732453+0000","last_change":"2024-09-15T20:29:33.126879+0000","last_active":"2024-09-15T20:29:49.732453+0000","last_peered":"2024-09-15T20:29:49.732453+0000","last_clean":"2024-09-15T20:29:49.732453+0000","last_became_active":"2024-09-15T20:29:33.126528+0000","last_became_peered":"2024-09-15T20:29:33.126528+0000","last_unstale":"2024-09-15T20:29:49.732453+0000","last_undegraded":"2024-09-15T20:29:49.732453+0000","last_fullsized":"2024-09-15T20:29:49.732453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:58.496790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640315,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709040,"kb_used_data":627868,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662800,"statfs":{"total":96636764160,"available":95910707200,"internally_reserved":0,"allocated":642936832,"data_stored":10282462078,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95500,"kb_used_data":19896,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276340,"statfs":{"total":96636764160,"available":96538972160,"internally_reserved":0,"allocated":20373504,"data_stored":321448830,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":311296,"data_stored":449918,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:29:55.897 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'22860", 'reported_seq': 22922, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2024-09-15T20:29:49.229908+0000', 'last_change': '2024-09-15T20:29:34.150898+0000', 'last_active': '2024-09-15T20:29:49.229908+0000', 'last_peered': '2024-09-15T20:29:49.229908+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:29:34.149984+0000', 'last_became_peered': '2024-09-15T20:29:34.149984+0000', 'last_unstale': '2024-09-15T20:29:49.229908+0000', 'last_undegraded': '2024-09-15T20:29:34.131101+0000', 'last_fullsized': '2024-09-15T20:29:34.128409+0000', 'mapping_epoch': 29, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:29:55.898 DEBUG:tasks.backfill_toofull:not backfilling 2024-09-15T20:30:00.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T20:30:00.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T20:30:00.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T20:30:00.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T20:30:01.058 INFO:teuthology.orchestra.run.smithi129.stdout:47244640317 2024-09-15T20:30:01.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:30:01.059 INFO:teuthology.orchestra.run.smithi129.stdout:120259084298 2024-09-15T20:30:01.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:30:01.061 INFO:teuthology.orchestra.run.smithi129.stdout:38654705724 2024-09-15T20:30:01.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:30:01.062 INFO:teuthology.orchestra.run.smithi129.stdout:38654705724 2024-09-15T20:30:01.062 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:30:01.404 INFO:teuthology.orchestra.run.smithi129.stdout:120259084297 2024-09-15T20:30:01.412 INFO:teuthology.orchestra.run.smithi129.stdout:38654705723 2024-09-15T20:30:01.418 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084298 got 120259084297 for osd.1 2024-09-15T20:30:01.430 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705724 got 38654705723 for osd.2 2024-09-15T20:30:01.452 INFO:teuthology.orchestra.run.smithi129.stdout:38654705723 2024-09-15T20:30:01.460 INFO:teuthology.orchestra.run.smithi129.stdout:47244640316 2024-09-15T20:30:01.470 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705724 got 38654705723 for osd.0 2024-09-15T20:30:01.476 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640317 got 47244640316 for osd.3 2024-09-15T20:30:02.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:30:02.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:30:02.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:30:02.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:30:02.742 INFO:teuthology.orchestra.run.smithi129.stdout:120259084298 2024-09-15T20:30:02.761 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084298 got 120259084298 for osd.1 2024-09-15T20:30:02.761 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:02.783 INFO:teuthology.orchestra.run.smithi129.stdout:38654705724 2024-09-15T20:30:02.796 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705724 got 38654705724 for osd.2 2024-09-15T20:30:02.797 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:02.825 INFO:teuthology.orchestra.run.smithi129.stdout:38654705724 2024-09-15T20:30:02.827 INFO:teuthology.orchestra.run.smithi129.stdout:47244640318 2024-09-15T20:30:02.839 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705724 got 38654705724 for osd.0 2024-09-15T20:30:02.839 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:02.840 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640317 got 47244640318 for osd.3 2024-09-15T20:30:02.840 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:02.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:03.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:03.159 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:03.172 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-15T20:30:01.602315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2019984,"kb_used_data":1797428,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467376,"statfs":{"total":386547056640,"available":384478593024,"internally_reserved":0,"allocated":1840566272,"data_stored":29430944248,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002487"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22922,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-15T20:29:49.229908+0000","last_change":"2024-09-15T20:29:34.150898+0000","last_active":"2024-09-15T20:29:49.229908+0000","last_peered":"2024-09-15T20:29:49.229908+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:29:34.149984+0000","last_became_peered":"2024-09-15T20:29:34.149984+0000","last_unstale":"2024-09-15T20:29:49.229908+0000","last_undegraded":"2024-09-15T20:29:34.131101+0000","last_fullsized":"2024-09-15T20:29:34.128409+0000","mapping_epoch":29,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T20:29:49.732453+0000","last_change":"2024-09-15T20:29:33.126879+0000","last_active":"2024-09-15T20:29:49.732453+0000","last_peered":"2024-09-15T20:29:49.732453+0000","last_clean":"2024-09-15T20:29:49.732453+0000","last_became_active":"2024-09-15T20:29:33.126528+0000","last_became_peered":"2024-09-15T20:29:33.126528+0000","last_unstale":"2024-09-15T20:29:49.732453+0000","last_undegraded":"2024-09-15T20:29:49.732453+0000","last_fullsized":"2024-09-15T20:29:49.732453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:58.496790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640318,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709040,"kb_used_data":627868,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662800,"statfs":{"total":96636764160,"available":95910707200,"internally_reserved":0,"allocated":642936832,"data_stored":10282462078,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95500,"kb_used_data":19896,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276340,"statfs":{"total":96636764160,"available":96538972160,"internally_reserved":0,"allocated":20373504,"data_stored":321448830,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":311296,"data_stored":449918,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:03.173 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'22860", 'reported_seq': 22922, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2024-09-15T20:29:49.229908+0000', 'last_change': '2024-09-15T20:29:34.150898+0000', 'last_active': '2024-09-15T20:29:49.229908+0000', 'last_peered': '2024-09-15T20:29:49.229908+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:29:34.149984+0000', 'last_became_peered': '2024-09-15T20:29:34.149984+0000', 'last_unstale': '2024-09-15T20:29:49.229908+0000', 'last_undegraded': '2024-09-15T20:29:34.131101+0000', 'last_fullsized': '2024-09-15T20:29:34.128409+0000', 'mapping_epoch': 29, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:03.173 DEBUG:tasks.backfill_toofull:not backfilling 2024-09-15T20:30:08.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T20:30:08.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T20:30:08.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T20:30:08.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T20:30:08.338 INFO:teuthology.orchestra.run.smithi129.stdout:47244640320 2024-09-15T20:30:08.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:30:08.340 INFO:teuthology.orchestra.run.smithi129.stdout:120259084300 2024-09-15T20:30:08.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:30:08.350 INFO:teuthology.orchestra.run.smithi129.stdout:38654705726 2024-09-15T20:30:08.350 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:30:08.352 INFO:teuthology.orchestra.run.smithi129.stdout:38654705726 2024-09-15T20:30:08.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:30:08.656 INFO:teuthology.orchestra.run.smithi129.stdout:47244640319 2024-09-15T20:30:08.669 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640320 got 47244640319 for osd.3 2024-09-15T20:30:08.693 INFO:teuthology.orchestra.run.smithi129.stdout:38654705725 2024-09-15T20:30:08.698 INFO:teuthology.orchestra.run.smithi129.stdout:38654705725 2024-09-15T20:30:08.698 INFO:teuthology.orchestra.run.smithi129.stdout:120259084299 2024-09-15T20:30:08.708 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705726 got 38654705725 for osd.2 2024-09-15T20:30:08.714 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705726 got 38654705725 for osd.0 2024-09-15T20:30:08.714 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084300 got 120259084299 for osd.1 2024-09-15T20:30:09.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:30:09.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:30:09.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:30:09.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:30:09.991 INFO:teuthology.orchestra.run.smithi129.stdout:47244640319 2024-09-15T20:30:10.008 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640320 got 47244640319 for osd.3 2024-09-15T20:30:10.023 INFO:teuthology.orchestra.run.smithi129.stdout:38654705725 2024-09-15T20:30:10.025 INFO:teuthology.orchestra.run.smithi129.stdout:38654705725 2024-09-15T20:30:10.038 INFO:teuthology.orchestra.run.smithi129.stdout:120259084299 2024-09-15T20:30:10.040 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705726 got 38654705725 for osd.0 2024-09-15T20:30:10.042 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705726 got 38654705725 for osd.2 2024-09-15T20:30:10.052 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084300 got 120259084299 for osd.1 2024-09-15T20:30:11.009 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:30:11.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:30:11.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:30:11.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:30:11.335 INFO:teuthology.orchestra.run.smithi129.stdout:47244640320 2024-09-15T20:30:11.348 INFO:tasks.backfill_toofull.ceph_manager:need seq 47244640320 got 47244640320 for osd.3 2024-09-15T20:30:11.349 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:11.370 INFO:teuthology.orchestra.run.smithi129.stdout:38654705727 2024-09-15T20:30:11.384 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705726 got 38654705727 for osd.2 2024-09-15T20:30:11.384 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:11.407 INFO:teuthology.orchestra.run.smithi129.stdout:120259084301 2024-09-15T20:30:11.412 INFO:teuthology.orchestra.run.smithi129.stdout:38654705727 2024-09-15T20:30:11.424 INFO:tasks.backfill_toofull.ceph_manager:need seq 120259084300 got 120259084301 for osd.1 2024-09-15T20:30:11.424 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:11.429 INFO:tasks.backfill_toofull.ceph_manager:need seq 38654705726 got 38654705727 for osd.0 2024-09-15T20:30:11.429 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:11.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:11.752 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:11.752 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:11.764 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-15T20:30:11.604322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2019984,"kb_used_data":1797428,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467376,"statfs":{"total":386547056640,"available":384478593024,"internally_reserved":0,"allocated":1840566272,"data_stored":29430944248,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002500"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22925,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:04.151311+0000","last_change":"2024-09-15T20:30:04.151804+0000","last_active":"2024-09-15T20:30:04.151311+0000","last_peered":"2024-09-15T20:30:04.151311+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:29:34.149984+0000","last_became_peered":"2024-09-15T20:29:34.149984+0000","last_unstale":"2024-09-15T20:30:04.151311+0000","last_undegraded":"2024-09-15T20:29:34.131101+0000","last_fullsized":"2024-09-15T20:29:34.128409+0000","mapping_epoch":29,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T20:29:49.732453+0000","last_change":"2024-09-15T20:29:33.126879+0000","last_active":"2024-09-15T20:29:49.732453+0000","last_peered":"2024-09-15T20:29:49.732453+0000","last_clean":"2024-09-15T20:29:49.732453+0000","last_became_active":"2024-09-15T20:29:33.126528+0000","last_became_peered":"2024-09-15T20:29:33.126528+0000","last_unstale":"2024-09-15T20:29:49.732453+0000","last_undegraded":"2024-09-15T20:29:49.732453+0000","last_fullsized":"2024-09-15T20:29:49.732453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:58.496790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709040,"kb_used_data":627868,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662800,"statfs":{"total":96636764160,"available":95910707200,"internally_reserved":0,"allocated":642936832,"data_stored":10282462078,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95500,"kb_used_data":19896,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276340,"statfs":{"total":96636764160,"available":96538972160,"internally_reserved":0,"allocated":20373504,"data_stored":321448830,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":311296,"data_stored":449918,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:11.765 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'22860", 'reported_seq': 22925, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:04.151311+0000', 'last_change': '2024-09-15T20:30:04.151804+0000', 'last_active': '2024-09-15T20:30:04.151311+0000', 'last_peered': '2024-09-15T20:30:04.151311+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:29:34.149984+0000', 'last_became_peered': '2024-09-15T20:29:34.149984+0000', 'last_unstale': '2024-09-15T20:30:04.151311+0000', 'last_undegraded': '2024-09-15T20:29:34.131101+0000', 'last_fullsized': '2024-09-15T20:29:34.128409+0000', 'mapping_epoch': 29, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:11.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:12.043 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:12.043 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:12.055 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-15T20:30:11.604322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2019984,"kb_used_data":1797428,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467376,"statfs":{"total":386547056640,"available":384478593024,"internally_reserved":0,"allocated":1840566272,"data_stored":29430944248,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002500"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22925,"reported_epoch":34,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:04.151311+0000","last_change":"2024-09-15T20:30:04.151804+0000","last_active":"2024-09-15T20:30:04.151311+0000","last_peered":"2024-09-15T20:30:04.151311+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:29:34.149984+0000","last_became_peered":"2024-09-15T20:29:34.149984+0000","last_unstale":"2024-09-15T20:30:04.151311+0000","last_undegraded":"2024-09-15T20:29:34.131101+0000","last_fullsized":"2024-09-15T20:29:34.128409+0000","mapping_epoch":29,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-15T20:29:49.732453+0000","last_change":"2024-09-15T20:29:33.126879+0000","last_active":"2024-09-15T20:29:49.732453+0000","last_peered":"2024-09-15T20:29:49.732453+0000","last_clean":"2024-09-15T20:29:49.732453+0000","last_became_active":"2024-09-15T20:29:33.126528+0000","last_became_peered":"2024-09-15T20:29:33.126528+0000","last_unstale":"2024-09-15T20:29:49.732453+0000","last_undegraded":"2024-09-15T20:29:49.732453+0000","last_fullsized":"2024-09-15T20:29:49.732453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:58.496790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709040,"kb_used_data":627868,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662800,"statfs":{"total":96636764160,"available":95910707200,"internally_reserved":0,"allocated":642936832,"data_stored":10282462078,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95500,"kb_used_data":19896,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276340,"statfs":{"total":96636764160,"available":96538972160,"internally_reserved":0,"allocated":20373504,"data_stored":321448830,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":311296,"data_stored":449918,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:12.056 DEBUG:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'22860", 'reported_seq': 22925, 'reported_epoch': 34, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:04.151311+0000', 'last_change': '2024-09-15T20:30:04.151804+0000', 'last_active': '2024-09-15T20:30:04.151311+0000', 'last_peered': '2024-09-15T20:30:04.151311+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:29:34.149984+0000', 'last_became_peered': '2024-09-15T20:29:34.149984+0000', 'last_unstale': '2024-09-15T20:30:04.151311+0000', 'last_undegraded': '2024-09-15T20:29:34.131101+0000', 'last_fullsized': '2024-09-15T20:29:34.128409+0000', 'mapping_epoch': 29, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:12.056 DEBUG:tasks.backfill_toofull:interrupt 1 backfill 2024-09-15T20:30:12.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-15T20:30:12.351 INFO:tasks.ceph.mon.c.smithi129.stderr:2024-09-15T20:30:12.348+0000 7fcd48a05640 -1 mon.c@2(leader).osd e34 definitely_dead 0 2024-09-15T20:30:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:marked down osd.1. 2024-09-15T20:30:13.360 DEBUG:tasks.backfill_toofull:resume recovery 2024-09-15T20:30:13.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noout 2024-09-15T20:30:14.347 INFO:teuthology.orchestra.run.smithi129.stderr:noout is unset 2024-09-15T20:30:14.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nobackfill 2024-09-15T20:30:15.345 INFO:teuthology.orchestra.run.smithi129.stderr:nobackfill is unset 2024-09-15T20:30:15.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset norecover 2024-09-15T20:30:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:norecover is unset 2024-09-15T20:30:16.376 INFO:tasks.backfill_toofull.ceph_manager:waiting for clean 2024-09-15T20:30:16.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:16.662 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:16.663 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:16.675 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-15T20:30:16.356582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2020020,"kb_used_data":1797460,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467340,"statfs":{"total":386547056640,"available":384478556160,"internally_reserved":0,"allocated":1840599040,"data_stored":29430960386,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.752259"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22935,"reported_epoch":37,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-15T20:30:15.456770+0000","last_change":"2024-09-15T20:30:14.348005+0000","last_active":"2024-09-15T20:30:15.456770+0000","last_peered":"2024-09-15T20:30:15.456770+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:14.348005+0000","last_became_peered":"2024-09-15T20:30:14.348005+0000","last_unstale":"2024-09-15T20:30:15.456770+0000","last_undegraded":"2024-09-15T20:30:14.345472+0000","last_fullsized":"2024-09-15T20:30:14.345299+0000","mapping_epoch":35,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":35,"state":"peering","last_fresh":"2024-09-15T20:30:13.343304+0000","last_change":"2024-09-15T20:30:13.343304+0000","last_active":"2024-09-15T20:30:13.343304+0000","last_peered":"2024-09-15T20:29:29.290056+0000","last_clean":"2024-09-15T20:25:35.263838+0000","last_became_active":"2024-09-15T20:27:59.832461+0000","last_became_peered":"2024-09-15T20:27:59.832461+0000","last_unstale":"2024-09-15T20:30:13.343304+0000","last_undegraded":"2024-09-15T20:30:13.343304+0000","last_fullsized":"2024-09-15T20:30:13.343304+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709060,"kb_used_data":627884,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662780,"statfs":{"total":96636764160,"available":95910686720,"internally_reserved":0,"allocated":642953216,"data_stored":10282470130,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95508,"kb_used_data":19904,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276332,"statfs":{"total":96636764160,"available":96538963968,"internally_reserved":0,"allocated":20381696,"data_stored":321452873,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":453961,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:16.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:16.957 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:16.957 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:16.970 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-15T20:30:16.356582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2020020,"kb_used_data":1797460,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467340,"statfs":{"total":386547056640,"available":384478556160,"internally_reserved":0,"allocated":1840599040,"data_stored":29430960386,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.752259"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22935,"reported_epoch":37,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-15T20:30:15.456770+0000","last_change":"2024-09-15T20:30:14.348005+0000","last_active":"2024-09-15T20:30:15.456770+0000","last_peered":"2024-09-15T20:30:15.456770+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:14.348005+0000","last_became_peered":"2024-09-15T20:30:14.348005+0000","last_unstale":"2024-09-15T20:30:15.456770+0000","last_undegraded":"2024-09-15T20:30:14.345472+0000","last_fullsized":"2024-09-15T20:30:14.345299+0000","mapping_epoch":35,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":35,"state":"peering","last_fresh":"2024-09-15T20:30:13.343304+0000","last_change":"2024-09-15T20:30:13.343304+0000","last_active":"2024-09-15T20:30:13.343304+0000","last_peered":"2024-09-15T20:29:29.290056+0000","last_clean":"2024-09-15T20:25:35.263838+0000","last_became_active":"2024-09-15T20:27:59.832461+0000","last_became_peered":"2024-09-15T20:27:59.832461+0000","last_unstale":"2024-09-15T20:30:13.343304+0000","last_undegraded":"2024-09-15T20:30:13.343304+0000","last_fullsized":"2024-09-15T20:30:13.343304+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709060,"kb_used_data":627884,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662780,"statfs":{"total":96636764160,"available":95910686720,"internally_reserved":0,"allocated":642953216,"data_stored":10282470130,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95508,"kb_used_data":19904,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276332,"statfs":{"total":96636764160,"available":96538963968,"internally_reserved":0,"allocated":20381696,"data_stored":321452873,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":453961,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:16.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:17.252 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:17.265 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-15T20:30:16.356582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2020020,"kb_used_data":1797460,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467340,"statfs":{"total":386547056640,"available":384478556160,"internally_reserved":0,"allocated":1840599040,"data_stored":29430960386,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.752259"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22935,"reported_epoch":37,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-15T20:30:15.456770+0000","last_change":"2024-09-15T20:30:14.348005+0000","last_active":"2024-09-15T20:30:15.456770+0000","last_peered":"2024-09-15T20:30:15.456770+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:14.348005+0000","last_became_peered":"2024-09-15T20:30:14.348005+0000","last_unstale":"2024-09-15T20:30:15.456770+0000","last_undegraded":"2024-09-15T20:30:14.345472+0000","last_fullsized":"2024-09-15T20:30:14.345299+0000","mapping_epoch":35,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":35,"state":"peering","last_fresh":"2024-09-15T20:30:13.343304+0000","last_change":"2024-09-15T20:30:13.343304+0000","last_active":"2024-09-15T20:30:13.343304+0000","last_peered":"2024-09-15T20:29:29.290056+0000","last_clean":"2024-09-15T20:25:35.263838+0000","last_became_active":"2024-09-15T20:27:59.832461+0000","last_became_peered":"2024-09-15T20:27:59.832461+0000","last_unstale":"2024-09-15T20:30:13.343304+0000","last_undegraded":"2024-09-15T20:30:13.343304+0000","last_fullsized":"2024-09-15T20:30:13.343304+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709060,"kb_used_data":627884,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662780,"statfs":{"total":96636764160,"available":95910686720,"internally_reserved":0,"allocated":642953216,"data_stored":10282470130,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95508,"kb_used_data":19904,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276332,"statfs":{"total":96636764160,"available":96538963968,"internally_reserved":0,"allocated":20381696,"data_stored":321452873,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":453961,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:17.265 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:17.265 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 22935, 'reported_epoch': 37, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2024-09-15T20:30:15.456770+0000', 'last_change': '2024-09-15T20:30:14.348005+0000', 'last_active': '2024-09-15T20:30:15.456770+0000', 'last_peered': '2024-09-15T20:30:15.456770+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:14.348005+0000', 'last_became_peered': '2024-09-15T20:30:14.348005+0000', 'last_unstale': '2024-09-15T20:30:15.456770+0000', 'last_undegraded': '2024-09-15T20:30:14.345472+0000', 'last_fullsized': '2024-09-15T20:30:14.345299+0000', 'mapping_epoch': 35, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2147483647, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:17.265 INFO:tasks.backfill_toofull.ceph_manager:PG 1.0 is not active+clean 2024-09-15T20:30:17.266 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '1.0', 'version': "11'17", 'reported_seq': 50, 'reported_epoch': 35, 'state': 'peering', 'last_fresh': '2024-09-15T20:30:13.343304+0000', 'last_change': '2024-09-15T20:30:13.343304+0000', 'last_active': '2024-09-15T20:30:13.343304+0000', 'last_peered': '2024-09-15T20:29:29.290056+0000', 'last_clean': '2024-09-15T20:25:35.263838+0000', 'last_became_active': '2024-09-15T20:27:59.832461+0000', 'last_became_peered': '2024-09-15T20:27:59.832461+0000', 'last_unstale': '2024-09-15T20:30:13.343304+0000', 'last_undegraded': '2024-09-15T20:30:13.343304+0000', 'last_fullsized': '2024-09-15T20:30:13.343304+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 29, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:33.583068+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:33.583068+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:33.583068+0000', 'objects_scrubbed': 0, 'log_size': 17, 'log_dups_size': 0, 'ondisk_log_size': 17, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 328192, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 26, 'num_read_kb': 19, 'num_write': 37, 'num_write_kb': 322, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T20:30:17.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:17.616 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:17.617 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4904/14716 objects degraded (33.324%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":294,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":3,"osd_up_since":1726432213,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"peering","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":2068480000,"bytes_avail":384478576640,"bytes_total":386547056640,"inactive_pgs_ratio":0.5,"degraded_objects":4904,"degraded_total":14716,"degraded_ratio":0.33324272900244639},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-15T20:26:37.568593+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:17.629 INFO:tasks.backfill_toofull.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T20:30:17.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:17.757 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-15T20:30:17.754+0000 7f0db7b5e640 -1 osd.1 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:30:17.908 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:17.908 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:17.921 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-15T20:30:17.605234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":4,"acting":4,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2020020,"kb_used_data":1797460,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467340,"statfs":{"total":386547056640,"available":384478556160,"internally_reserved":0,"allocated":1840599040,"data_stored":29430960386,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000624"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22935,"reported_epoch":37,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-15T20:30:15.456770+0000","last_change":"2024-09-15T20:30:14.348005+0000","last_active":"2024-09-15T20:30:15.456770+0000","last_peered":"2024-09-15T20:30:15.456770+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:14.348005+0000","last_became_peered":"2024-09-15T20:30:14.348005+0000","last_unstale":"2024-09-15T20:30:15.456770+0000","last_undegraded":"2024-09-15T20:30:14.345472+0000","last_fullsized":"2024-09-15T20:30:14.345299+0000","mapping_epoch":35,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":35,"state":"peering","last_fresh":"2024-09-15T20:30:13.343304+0000","last_change":"2024-09-15T20:30:13.343304+0000","last_active":"2024-09-15T20:30:13.343304+0000","last_peered":"2024-09-15T20:29:29.290056+0000","last_clean":"2024-09-15T20:25:35.263838+0000","last_became_active":"2024-09-15T20:27:59.832461+0000","last_became_peered":"2024-09-15T20:27:59.832461+0000","last_unstale":"2024-09-15T20:30:13.343304+0000","last_undegraded":"2024-09-15T20:30:13.343304+0000","last_fullsized":"2024-09-15T20:30:13.343304+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709060,"kb_used_data":627884,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662780,"statfs":{"total":96636764160,"available":95910686720,"internally_reserved":0,"allocated":642953216,"data_stored":10282470130,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95508,"kb_used_data":19904,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276332,"statfs":{"total":96636764160,"available":96538963968,"internally_reserved":0,"allocated":20381696,"data_stored":321452873,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188112,"kb_used_data":1149360,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183728,"statfs":{"total":96636764160,"available":95420137472,"internally_reserved":0,"allocated":1176944640,"data_stored":18826583422,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":453961,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:20.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:21.249 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:21.250 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:21.264 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-15T20:30:20.385401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2020160,"kb_used_data":1797588,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467200,"statfs":{"total":386547056640,"available":384478412800,"internally_reserved":0,"allocated":1840730112,"data_stored":29431025782,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028818"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22935,"reported_epoch":37,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-15T20:30:15.456770+0000","last_change":"2024-09-15T20:30:14.348005+0000","last_active":"2024-09-15T20:30:15.456770+0000","last_peered":"2024-09-15T20:30:15.456770+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:14.348005+0000","last_became_peered":"2024-09-15T20:30:14.348005+0000","last_unstale":"2024-09-15T20:30:15.456770+0000","last_undegraded":"2024-09-15T20:30:14.345472+0000","last_fullsized":"2024-09-15T20:30:14.345299+0000","mapping_epoch":35,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":69,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T20:30:19.365376+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:19.365376+0000","last_peered":"2024-09-15T20:30:19.365376+0000","last_clean":"2024-09-15T20:30:19.365376+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:19.365376+0000","last_undegraded":"2024-09-15T20:30:19.365376+0000","last_fullsized":"2024-09-15T20:30:19.365376+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709060,"kb_used_data":627884,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662780,"statfs":{"total":96636764160,"available":95910686720,"internally_reserved":0,"allocated":642953216,"data_stored":10282470130,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95556,"kb_used_data":19944,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276284,"statfs":{"total":96636764160,"available":96538914816,"internally_reserved":0,"allocated":20422656,"data_stored":321473324,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188156,"kb_used_data":1149408,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183684,"statfs":{"total":96636764160,"available":95420092416,"internally_reserved":0,"allocated":1176993792,"data_stored":18826607916,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":360448,"data_stored":474412,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:21.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:21.597 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:21.597 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:21.611 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-15T20:30:20.385401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2020160,"kb_used_data":1797588,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467200,"statfs":{"total":386547056640,"available":384478412800,"internally_reserved":0,"allocated":1840730112,"data_stored":29431025782,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028818"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":22935,"reported_epoch":37,"state":"active+undersized+degraded+wait","last_fresh":"2024-09-15T20:30:15.456770+0000","last_change":"2024-09-15T20:30:14.348005+0000","last_active":"2024-09-15T20:30:15.456770+0000","last_peered":"2024-09-15T20:30:15.456770+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:14.348005+0000","last_became_peered":"2024-09-15T20:30:14.348005+0000","last_unstale":"2024-09-15T20:30:15.456770+0000","last_undegraded":"2024-09-15T20:30:14.345472+0000","last_fullsized":"2024-09-15T20:30:14.345299+0000","mapping_epoch":35,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":69,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T20:30:19.365376+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:19.365376+0000","last_peered":"2024-09-15T20:30:19.365376+0000","last_clean":"2024-09-15T20:30:19.365376+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:19.365376+0000","last_undegraded":"2024-09-15T20:30:19.365376+0000","last_fullsized":"2024-09-15T20:30:19.365376+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709060,"kb_used_data":627884,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662780,"statfs":{"total":96636764160,"available":95910686720,"internally_reserved":0,"allocated":642953216,"data_stored":10282470130,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95556,"kb_used_data":19944,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276284,"statfs":{"total":96636764160,"available":96538914816,"internally_reserved":0,"allocated":20422656,"data_stored":321473324,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188156,"kb_used_data":1149408,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183684,"statfs":{"total":96636764160,"available":95420092416,"internally_reserved":0,"allocated":1176993792,"data_stored":18826607916,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":360448,"data_stored":474412,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:21.612 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:21.612 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 22935, 'reported_epoch': 37, 'state': 'active+undersized+degraded+wait', 'last_fresh': '2024-09-15T20:30:15.456770+0000', 'last_change': '2024-09-15T20:30:14.348005+0000', 'last_active': '2024-09-15T20:30:15.456770+0000', 'last_peered': '2024-09-15T20:30:15.456770+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:14.348005+0000', 'last_became_peered': '2024-09-15T20:30:14.348005+0000', 'last_unstale': '2024-09-15T20:30:15.456770+0000', 'last_undegraded': '2024-09-15T20:30:14.345472+0000', 'last_fullsized': '2024-09-15T20:30:14.345299+0000', 'mapping_epoch': 35, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T08:07:52.042754+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2147483647, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:21.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:22.015 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:22.015 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4904/14716 objects degraded (33.324%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":298,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+wait","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":2068643840,"bytes_avail":384478412800,"bytes_total":386547056640,"degraded_objects":4904,"degraded_total":14716,"degraded_ratio":0.33324272900244639},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-15T20:26:37.568593+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:22.029 INFO:tasks.backfill_toofull.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T20:30:22.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:22.355 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:22.355 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:22.369 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-15T20:30:21.606119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2020196,"kb_used_data":1797628,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375467164,"statfs":{"total":386547056640,"available":384478375936,"internally_reserved":0,"allocated":1840771072,"data_stored":29431046299,"data_compressed":42956122,"data_compressed_allocated":1839403008,"data_compressed_original":29430448128,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.253583"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23019,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:21.179559+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:21.179559+0000","last_peered":"2024-09-15T20:30:21.179559+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:21.179559+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":69,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T20:30:19.365376+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:19.365376+0000","last_peered":"2024-09-15T20:30:19.365376+0000","last_clean":"2024-09-15T20:30:19.365376+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:19.365376+0000","last_undegraded":"2024-09-15T20:30:19.365376+0000","last_fullsized":"2024-09-15T20:30:19.365376+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1839370240,"data_stored":29429800960,"data_compressed":42954302,"data_compressed_allocated":1839362048,"data_compressed_original":29429792768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640323,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709096,"kb_used_data":627924,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662744,"statfs":{"total":96636764160,"available":95910649856,"internally_reserved":0,"allocated":642994176,"data_stored":10282490647,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95556,"kb_used_data":19944,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276284,"statfs":{"total":96636764160,"available":96538914816,"internally_reserved":0,"allocated":20422656,"data_stored":321473324,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188156,"kb_used_data":1149408,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93183684,"statfs":{"total":96636764160,"available":95420092416,"internally_reserved":0,"allocated":1176993792,"data_stored":18826607916,"data_compressed":27290990,"data_compressed_allocated":1176653824,"data_compressed_original":18826461184,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":360448,"data_stored":474412,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176633344,"data_stored":18826133504,"data_compressed":27290080,"data_compressed_allocated":1176633344,"data_compressed_original":18826133504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:25.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:25.700 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:25.700 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:25.713 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-15T20:30:25.606737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2007168,"kb_used_data":1780460,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375480192,"statfs":{"total":386547056640,"available":384491716608,"internally_reserved":0,"allocated":1823191040,"data_stored":29149003682,"data_compressed":42547242,"data_compressed_allocated":1821773824,"data_compressed_original":29148381184,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.249863"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23019,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:21.179559+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:21.179559+0000","last_peered":"2024-09-15T20:30:21.179559+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:21.179559+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1814990848,"data_stored":29039730688,"data_compressed":42388862,"data_compressed_allocated":1814982656,"data_compressed_original":29039722496,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640323,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709096,"kb_used_data":627924,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662744,"statfs":{"total":96636764160,"available":95910649856,"internally_reserved":0,"allocated":642994176,"data_stored":10282490647,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1175104,"kb_used_data":1132208,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93196736,"statfs":{"total":96636764160,"available":95433457664,"internally_reserved":0,"allocated":1159380992,"data_stored":18544549081,"data_compressed":26882110,"data_compressed_allocated":1159024640,"data_compressed_original":18544394240,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1152253952,"data_stored":18436063232,"data_compressed":26724640,"data_compressed_allocated":1152253952,"data_compressed_original":18436063232,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:25.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:26.040 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:26.040 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:26.053 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-15T20:30:25.606737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2007168,"kb_used_data":1780460,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375480192,"statfs":{"total":386547056640,"available":384491716608,"internally_reserved":0,"allocated":1823191040,"data_stored":29149003682,"data_compressed":42547242,"data_compressed_allocated":1821773824,"data_compressed_original":29148381184,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.249863"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23019,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:21.179559+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:21.179559+0000","last_peered":"2024-09-15T20:30:21.179559+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:21.179559+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1814990848,"data_stored":29039730688,"data_compressed":42388862,"data_compressed_allocated":1814982656,"data_compressed_original":29039722496,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640323,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709096,"kb_used_data":627924,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662744,"statfs":{"total":96636764160,"available":95910649856,"internally_reserved":0,"allocated":642994176,"data_stored":10282490647,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1175104,"kb_used_data":1132208,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93196736,"statfs":{"total":96636764160,"available":95433457664,"internally_reserved":0,"allocated":1159380992,"data_stored":18544549081,"data_compressed":26882110,"data_compressed_allocated":1159024640,"data_compressed_original":18544394240,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1152253952,"data_stored":18436063232,"data_compressed":26724640,"data_compressed_allocated":1152253952,"data_compressed_original":18436063232,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:26.054 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:26.054 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 23019, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:21.179559+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:21.179559+0000', 'last_peered': '2024-09-15T20:30:21.179559+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:21.179559+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4871, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 142606336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:26.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:26.476 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:26.476 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4871/14716 objects degraded (33.100%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":303,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":2055340032,"bytes_avail":384491716608,"bytes_total":386547056640,"degraded_objects":4871,"degraded_total":14716,"degraded_ratio":0.33100027181299269,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":19670208,"recovering_keys_per_sec":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-15T20:26:37.568593+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:26.490 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:26.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:26.811 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:26.825 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-15T20:30:25.606737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2007168,"kb_used_data":1780460,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375480192,"statfs":{"total":386547056640,"available":384491716608,"internally_reserved":0,"allocated":1823191040,"data_stored":29149003682,"data_compressed":42547242,"data_compressed_allocated":1821773824,"data_compressed_original":29148381184,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.249863"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23019,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:21.179559+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:21.179559+0000","last_peered":"2024-09-15T20:30:21.179559+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:21.179559+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1814990848,"data_stored":29039730688,"data_compressed":42388862,"data_compressed_allocated":1814982656,"data_compressed_original":29039722496,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640323,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709096,"kb_used_data":627924,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662744,"statfs":{"total":96636764160,"available":95910649856,"internally_reserved":0,"allocated":642994176,"data_stored":10282490647,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1175104,"kb_used_data":1132208,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93196736,"statfs":{"total":96636764160,"available":95433457664,"internally_reserved":0,"allocated":1159380992,"data_stored":18544549081,"data_compressed":26882110,"data_compressed_allocated":1159024640,"data_compressed_original":18544394240,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1152253952,"data_stored":18436063232,"data_compressed":26724640,"data_compressed_allocated":1152253952,"data_compressed_original":18436063232,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:29.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:30.151 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:30.151 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:30.164 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-15T20:30:29.607349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1973356,"kb_used_data":1754964,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375514004,"statfs":{"total":386547056640,"available":384526340096,"internally_reserved":0,"allocated":1797083136,"data_stored":28731150180,"data_compressed":41941522,"data_compressed_allocated":1795657728,"data_compressed_original":28730523648,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001742"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23501,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:26.184983+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:26.184983+0000","last_peered":"2024-09-15T20:30:26.184983+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:26.184983+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1792413696,"data_stored":28678496256,"data_compressed":41865222,"data_compressed_allocated":1792405504,"data_compressed_original":28678488064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141280,"kb_used_data":1106704,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93230560,"statfs":{"total":96636764160,"available":95468093440,"internally_reserved":0,"allocated":1133264896,"data_stored":18126691545,"data_compressed":26276390,"data_compressed_allocated":1132908544,"data_compressed_original":18126536704,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1129676800,"data_stored":18074828800,"data_compressed":26201000,"data_compressed_allocated":1129676800,"data_compressed_original":18074828800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:30.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:30.491 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:30.491 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:30.505 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-15T20:30:29.607349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1973356,"kb_used_data":1754964,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375514004,"statfs":{"total":386547056640,"available":384526340096,"internally_reserved":0,"allocated":1797083136,"data_stored":28731150180,"data_compressed":41941522,"data_compressed_allocated":1795657728,"data_compressed_original":28730523648,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001742"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23501,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:26.184983+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:26.184983+0000","last_peered":"2024-09-15T20:30:26.184983+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:26.184983+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1792413696,"data_stored":28678496256,"data_compressed":41865222,"data_compressed_allocated":1792405504,"data_compressed_original":28678488064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141280,"kb_used_data":1106704,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93230560,"statfs":{"total":96636764160,"available":95468093440,"internally_reserved":0,"allocated":1133264896,"data_stored":18126691545,"data_compressed":26276390,"data_compressed_allocated":1132908544,"data_compressed_original":18126536704,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1129676800,"data_stored":18074828800,"data_compressed":26201000,"data_compressed_allocated":1129676800,"data_compressed_original":18074828800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:30.505 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:30.506 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 23501, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:26.184983+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:26.184983+0000', 'last_peered': '2024-09-15T20:30:26.184983+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:26.184983+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4630, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 275, 'num_bytes_recovered': 1153433600, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:30.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:30.912 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:30.912 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4630/14716 objects degraded (31.462%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":2020716544,"bytes_avail":384526340096,"bytes_total":386547056640,"degraded_objects":4630,"degraded_total":14716,"degraded_ratio":0.31462353900516443,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":115323267,"recovering_keys_per_sec":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:30.926 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:30.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:31.260 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:31.261 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:31.274 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-15T20:30:29.607349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1973356,"kb_used_data":1754964,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375514004,"statfs":{"total":386547056640,"available":384526340096,"internally_reserved":0,"allocated":1797083136,"data_stored":28731150180,"data_compressed":41941522,"data_compressed_allocated":1795657728,"data_compressed_original":28730523648,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001742"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23501,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:26.184983+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:26.184983+0000","last_peered":"2024-09-15T20:30:26.184983+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:26.184983+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1153433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1792413696,"data_stored":28678496256,"data_compressed":41865222,"data_compressed_allocated":1792405504,"data_compressed_original":28678488064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141280,"kb_used_data":1106704,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93230560,"statfs":{"total":96636764160,"available":95468093440,"internally_reserved":0,"allocated":1133264896,"data_stored":18126691545,"data_compressed":26276390,"data_compressed_allocated":1132908544,"data_compressed_original":18126536704,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1129676800,"data_stored":18074828800,"data_compressed":26201000,"data_compressed_allocated":1129676800,"data_compressed_original":18074828800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:34.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:34.607 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:34.608 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:34.620 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-15T20:30:33.607988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1973356,"kb_used_data":1754964,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375514004,"statfs":{"total":386547056640,"available":384526340096,"internally_reserved":0,"allocated":1797083136,"data_stored":28731150180,"data_compressed":41941522,"data_compressed_allocated":1795657728,"data_compressed_original":28730523648,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":482,"num_bytes_recovered":2021654528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001869"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23983,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:31.193315+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:31.193315+0000","last_peered":"2024-09-15T20:30:31.193315+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:31.193315+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1792413696,"data_stored":28678496256,"data_compressed":41865222,"data_compressed_allocated":1792405504,"data_compressed_original":28678488064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141280,"kb_used_data":1106704,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93230560,"statfs":{"total":96636764160,"available":95468093440,"internally_reserved":0,"allocated":1133264896,"data_stored":18126691545,"data_compressed":26276390,"data_compressed_allocated":1132908544,"data_compressed_original":18126536704,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1129676800,"data_stored":18074828800,"data_compressed":26201000,"data_compressed_allocated":1129676800,"data_compressed_original":18074828800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:34.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:34.957 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:34.957 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:34.971 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-15T20:30:33.607988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1973356,"kb_used_data":1754964,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375514004,"statfs":{"total":386547056640,"available":384526340096,"internally_reserved":0,"allocated":1797083136,"data_stored":28731150180,"data_compressed":41941522,"data_compressed_allocated":1795657728,"data_compressed_original":28730523648,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":482,"num_bytes_recovered":2021654528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001869"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23983,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:31.193315+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:31.193315+0000","last_peered":"2024-09-15T20:30:31.193315+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:31.193315+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1792413696,"data_stored":28678496256,"data_compressed":41865222,"data_compressed_allocated":1792405504,"data_compressed_original":28678488064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141280,"kb_used_data":1106704,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93230560,"statfs":{"total":96636764160,"available":95468093440,"internally_reserved":0,"allocated":1133264896,"data_stored":18126691545,"data_compressed":26276390,"data_compressed_allocated":1132908544,"data_compressed_original":18126536704,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1129676800,"data_stored":18074828800,"data_compressed":26201000,"data_compressed_allocated":1129676800,"data_compressed_original":18074828800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:34.972 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:34.972 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 23983, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:31.193315+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:31.193315+0000', 'last_peered': '2024-09-15T20:30:31.193315+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:31.193315+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 516, 'num_bytes_recovered': 2164260864, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:34.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:35.375 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:35.375 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4389/14716 objects degraded (29.825%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":312,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":2020716544,"bytes_avail":384526340096,"bytes_total":386547056640,"degraded_objects":4389,"degraded_total":14716,"degraded_ratio":0.29824680619733623,"recovering_objects_per_sec":40,"recovering_bytes_per_sec":168444974,"recovering_keys_per_sec":0,"num_objects_recovered":482,"num_bytes_recovered":2021654528,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:35.390 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:35.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:35.714 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:35.714 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:35.728 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-15T20:30:35.608242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1951500,"kb_used_data":1728948,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375535860,"statfs":{"total":386547056640,"available":384548720640,"internally_reserved":0,"allocated":1770442752,"data_stored":28304904036,"data_compressed":41323642,"data_compressed_allocated":1769017344,"data_compressed_original":28304277504,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":482,"num_bytes_recovered":2021654528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001845"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":23983,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:31.193315+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:31.193315+0000","last_peered":"2024-09-15T20:30:31.193315+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:31.193315+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":2164260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1765806080,"data_stored":28252774400,"data_compressed":41248102,"data_compressed_allocated":1765797888,"data_compressed_original":28252766208,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119424,"kb_used_data":1080688,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93252416,"statfs":{"total":96636764160,"available":95490473984,"internally_reserved":0,"allocated":1106624512,"data_stored":17700445401,"data_compressed":25658510,"data_compressed_allocated":1106268160,"data_compressed_original":17700290560,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1103069184,"data_stored":17649106944,"data_compressed":25583880,"data_compressed_allocated":1103069184,"data_compressed_original":17649106944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:38.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:39.059 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:39.059 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:39.073 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-15T20:30:37.608511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1951500,"kb_used_data":1728948,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375535860,"statfs":{"total":386547056640,"available":384548720640,"internally_reserved":0,"allocated":1770442752,"data_stored":28304904036,"data_compressed":41323642,"data_compressed_allocated":1769017344,"data_compressed_original":28304277504,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":720,"num_bytes_recovered":3019898880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001773"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":24459,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:36.184696+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:36.184696+0000","last_peered":"2024-09-15T20:30:36.184696+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:36.184696+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1765806080,"data_stored":28252774400,"data_compressed":41248102,"data_compressed_allocated":1765797888,"data_compressed_original":28252766208,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640326,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119424,"kb_used_data":1080688,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93252416,"statfs":{"total":96636764160,"available":95490473984,"internally_reserved":0,"allocated":1106624512,"data_stored":17700445401,"data_compressed":25658510,"data_compressed_allocated":1106268160,"data_compressed_original":17700290560,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1103069184,"data_stored":17649106944,"data_compressed":25583880,"data_compressed_allocated":1103069184,"data_compressed_original":17649106944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:39.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:39.394 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:39.394 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:39.408 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-15T20:30:37.608511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1951500,"kb_used_data":1728948,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375535860,"statfs":{"total":386547056640,"available":384548720640,"internally_reserved":0,"allocated":1770442752,"data_stored":28304904036,"data_compressed":41323642,"data_compressed_allocated":1769017344,"data_compressed_original":28304277504,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":720,"num_bytes_recovered":3019898880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001773"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":24459,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:36.184696+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:36.184696+0000","last_peered":"2024-09-15T20:30:36.184696+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:36.184696+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1765806080,"data_stored":28252774400,"data_compressed":41248102,"data_compressed_allocated":1765797888,"data_compressed_original":28252766208,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640326,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119424,"kb_used_data":1080688,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93252416,"statfs":{"total":96636764160,"available":95490473984,"internally_reserved":0,"allocated":1106624512,"data_stored":17700445401,"data_compressed":25658510,"data_compressed_allocated":1106268160,"data_compressed_original":17700290560,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1103069184,"data_stored":17649106944,"data_compressed":25583880,"data_compressed_allocated":1103069184,"data_compressed_original":17649106944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:39.408 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:39.408 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 24459, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:36.184696+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:36.184696+0000', 'last_peered': '2024-09-15T20:30:36.184696+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:36.184696+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 754, 'num_bytes_recovered': 3162505216, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:39.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:39.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:39.824 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4151/14716 objects degraded (28.207%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":316,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1980313600,"bytes_avail":384566743040,"bytes_total":386547056640,"degraded_objects":4151,"degraded_total":14716,"degraded_ratio":0.28207393313400381,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":167398304,"recovering_keys_per_sec":0,"num_objects_recovered":479,"num_bytes_recovered":2009071616,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:39.837 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:39.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:40.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:40.160 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:40.172 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-15T20:30:39.608827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1933900,"kb_used_data":1707188,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375553460,"statfs":{"total":386547056640,"available":384566743040,"internally_reserved":0,"allocated":1748160512,"data_stored":27948388196,"data_compressed":40806842,"data_compressed_allocated":1746735104,"data_compressed_original":27947761664,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":2009071616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001744"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":24459,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:36.184696+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:36.184696+0000","last_peered":"2024-09-15T20:30:36.184696+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:36.184696+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":3162505216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1742999552,"data_stored":27887869952,"data_compressed":40719142,"data_compressed_allocated":1742991360,"data_compressed_original":27887861760,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640326,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1101824,"kb_used_data":1058928,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93270016,"statfs":{"total":96636764160,"available":95508496384,"internally_reserved":0,"allocated":1084342272,"data_stored":17343929561,"data_compressed":25141710,"data_compressed_allocated":1083985920,"data_compressed_original":17343774720,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1080262656,"data_stored":17284202496,"data_compressed":25054920,"data_compressed_allocated":1080262656,"data_compressed_original":17284202496,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:43.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:43.499 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:43.500 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:43.513 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-15T20:30:41.609112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1933900,"kb_used_data":1707188,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375553460,"statfs":{"total":386547056640,"available":384566743040,"internally_reserved":0,"allocated":1748160512,"data_stored":27948388196,"data_compressed":40806842,"data_compressed_allocated":1746735104,"data_compressed_original":27947761664,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":718,"num_bytes_recovered":3011510272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001763"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":24937,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:41.183155+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:41.183155+0000","last_peered":"2024-09-15T20:30:41.183155+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:41.183155+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1742999552,"data_stored":27887869952,"data_compressed":40719142,"data_compressed_allocated":1742991360,"data_compressed_original":27887861760,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640327,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1101824,"kb_used_data":1058928,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93270016,"statfs":{"total":96636764160,"available":95508496384,"internally_reserved":0,"allocated":1084342272,"data_stored":17343929561,"data_compressed":25141710,"data_compressed_allocated":1083985920,"data_compressed_original":17343774720,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1080262656,"data_stored":17284202496,"data_compressed":25054920,"data_compressed_allocated":1080262656,"data_compressed_original":17284202496,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:43.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:43.837 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:43.837 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:43.850 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-15T20:30:43.609406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1933900,"kb_used_data":1707188,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375553460,"statfs":{"total":386547056640,"available":384566743040,"internally_reserved":0,"allocated":1748160512,"data_stored":27948388196,"data_compressed":40806842,"data_compressed_allocated":1746735104,"data_compressed_original":27947761664,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":2000683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001737"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":24937,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:41.183155+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:41.183155+0000","last_peered":"2024-09-15T20:30:41.183155+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:41.183155+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1742999552,"data_stored":27887869952,"data_compressed":40719142,"data_compressed_allocated":1742991360,"data_compressed_original":27887861760,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640327,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1101824,"kb_used_data":1058928,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93270016,"statfs":{"total":96636764160,"available":95508496384,"internally_reserved":0,"allocated":1084342272,"data_stored":17343929561,"data_compressed":25141710,"data_compressed_allocated":1083985920,"data_compressed_original":17343774720,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1080262656,"data_stored":17284202496,"data_compressed":25054920,"data_compressed_allocated":1080262656,"data_compressed_original":17284202496,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:43.851 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:43.851 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 24937, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:41.183155+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:41.183155+0000', 'last_peered': '2024-09-15T20:30:41.183155+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:41.183155+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3912, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 993, 'num_bytes_recovered': 4164943872, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:43.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:44.261 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:44.261 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3912/14716 objects degraded (26.583%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":321,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1980313600,"bytes_avail":384566743040,"bytes_total":386547056640,"degraded_objects":3912,"degraded_total":14716,"degraded_ratio":0.2658331068225061,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":166699449,"recovering_keys_per_sec":0,"num_objects_recovered":477,"num_bytes_recovered":2000683008,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:44.275 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:44.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:44.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:44.606 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:44.618 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-15T20:30:43.609406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1933900,"kb_used_data":1707188,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375553460,"statfs":{"total":386547056640,"available":384566743040,"internally_reserved":0,"allocated":1748160512,"data_stored":27948388196,"data_compressed":40806842,"data_compressed_allocated":1746735104,"data_compressed_original":27947761664,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":2000683008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001737"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":24937,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:41.183155+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:41.183155+0000","last_peered":"2024-09-15T20:30:41.183155+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:41.183155+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":4164943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1742999552,"data_stored":27887869952,"data_compressed":40719142,"data_compressed_allocated":1742991360,"data_compressed_original":27887861760,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640327,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1101824,"kb_used_data":1058928,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93270016,"statfs":{"total":96636764160,"available":95508496384,"internally_reserved":0,"allocated":1084342272,"data_stored":17343929561,"data_compressed":25141710,"data_compressed_allocated":1083985920,"data_compressed_original":17343774720,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1080262656,"data_stored":17284202496,"data_compressed":25054920,"data_compressed_allocated":1080262656,"data_compressed_original":17284202496,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:47.620 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:47.950 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:47.950 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:47.963 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T20:30:47.610039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1907468,"kb_used_data":1689076,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375579892,"statfs":{"total":386547056640,"available":384593809408,"internally_reserved":0,"allocated":1729613824,"data_stored":27651641188,"data_compressed":40376682,"data_compressed_allocated":1728188416,"data_compressed_original":27651014656,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":2977955840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001797"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25403,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:46.194331+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:46.194331+0000","last_peered":"2024-09-15T20:30:46.194331+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:46.194331+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1727205376,"data_stored":27635163136,"data_compressed":40352822,"data_compressed_allocated":1727197184,"data_compressed_original":27635154944,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640328,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1075392,"kb_used_data":1040816,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93296448,"statfs":{"total":96636764160,"available":95535562752,"internally_reserved":0,"allocated":1065795584,"data_stored":17047182553,"data_compressed":24711550,"data_compressed_allocated":1065439232,"data_compressed_original":17047027712,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1064468480,"data_stored":17031495680,"data_compressed":24688600,"data_compressed_allocated":1064468480,"data_compressed_original":17031495680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:47.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:48.286 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:48.300 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T20:30:47.610039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1907468,"kb_used_data":1689076,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375579892,"statfs":{"total":386547056640,"available":384593809408,"internally_reserved":0,"allocated":1729613824,"data_stored":27651641188,"data_compressed":40376682,"data_compressed_allocated":1728188416,"data_compressed_original":27651014656,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":2977955840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001797"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25403,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:46.194331+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:46.194331+0000","last_peered":"2024-09-15T20:30:46.194331+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:46.194331+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1727205376,"data_stored":27635163136,"data_compressed":40352822,"data_compressed_allocated":1727197184,"data_compressed_original":27635154944,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640328,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1075392,"kb_used_data":1040816,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93296448,"statfs":{"total":96636764160,"available":95535562752,"internally_reserved":0,"allocated":1065795584,"data_stored":17047182553,"data_compressed":24711550,"data_compressed_allocated":1065439232,"data_compressed_original":17047027712,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1064468480,"data_stored":17031495680,"data_compressed":24688600,"data_compressed_allocated":1064468480,"data_compressed_original":17031495680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:48.300 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:48.300 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 25403, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:46.194331+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:46.194331+0000', 'last_peered': '2024-09-15T20:30:46.194331+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:46.194331+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3679, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1226, 'num_bytes_recovered': 5142216704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:48.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:48.702 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:48.702 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3679/14716 objects degraded (25.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1953247232,"bytes_avail":384593809408,"bytes_total":386547056640,"degraded_objects":3679,"degraded_total":14716,"degraded_ratio":0.25,"recovering_objects_per_sec":59,"recovering_bytes_per_sec":248125828,"recovering_keys_per_sec":0,"num_objects_recovered":710,"num_bytes_recovered":2977955840,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:48.716 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:48.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:49.037 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:49.051 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T20:30:47.610039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1907468,"kb_used_data":1689076,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375579892,"statfs":{"total":386547056640,"available":384593809408,"internally_reserved":0,"allocated":1729613824,"data_stored":27651641188,"data_compressed":40376682,"data_compressed_allocated":1728188416,"data_compressed_original":27651014656,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":2977955840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001797"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25403,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:46.194331+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:46.194331+0000","last_peered":"2024-09-15T20:30:46.194331+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:46.194331+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":5142216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1727205376,"data_stored":27635163136,"data_compressed":40352822,"data_compressed_allocated":1727197184,"data_compressed_original":27635154944,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640328,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1075392,"kb_used_data":1040816,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93296448,"statfs":{"total":96636764160,"available":95535562752,"internally_reserved":0,"allocated":1065795584,"data_stored":17047182553,"data_compressed":24711550,"data_compressed_allocated":1065439232,"data_compressed_original":17047027712,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1064468480,"data_stored":17031495680,"data_compressed":24688600,"data_compressed_allocated":1064468480,"data_compressed_original":17031495680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:52.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:52.375 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:52.389 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-15T20:30:51.610658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1894028,"kb_used_data":1671476,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375593332,"statfs":{"total":386547056640,"available":384607571968,"internally_reserved":0,"allocated":1711591424,"data_stored":27363282788,"data_compressed":39958682,"data_compressed_allocated":1710166016,"data_compressed_original":27362656256,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":711,"num_bytes_recovered":2982150144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001831"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25881,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:51.193752+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:51.193752+0000","last_peered":"2024-09-15T20:30:51.193752+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:51.193752+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1705873408,"data_stored":27293851648,"data_compressed":39858062,"data_compressed_allocated":1705865216,"data_compressed_original":27293843456,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640329,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1061952,"kb_used_data":1023216,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93309888,"statfs":{"total":96636764160,"available":95549325312,"internally_reserved":0,"allocated":1047773184,"data_stored":16758824153,"data_compressed":24293550,"data_compressed_allocated":1047416832,"data_compressed_original":16758669312,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1043136512,"data_stored":16690184192,"data_compressed":24193840,"data_compressed_allocated":1043136512,"data_compressed_original":16690184192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:52.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:52.714 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:52.715 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:52.729 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-15T20:30:51.610658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1894028,"kb_used_data":1671476,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375593332,"statfs":{"total":386547056640,"available":384607571968,"internally_reserved":0,"allocated":1711591424,"data_stored":27363282788,"data_compressed":39958682,"data_compressed_allocated":1710166016,"data_compressed_original":27362656256,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":711,"num_bytes_recovered":2982150144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001831"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25881,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:51.193752+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:51.193752+0000","last_peered":"2024-09-15T20:30:51.193752+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:51.193752+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1705873408,"data_stored":27293851648,"data_compressed":39858062,"data_compressed_allocated":1705865216,"data_compressed_original":27293843456,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640329,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1061952,"kb_used_data":1023216,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93309888,"statfs":{"total":96636764160,"available":95549325312,"internally_reserved":0,"allocated":1047773184,"data_stored":16758824153,"data_compressed":24293550,"data_compressed_allocated":1047416832,"data_compressed_original":16758669312,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1043136512,"data_stored":16690184192,"data_compressed":24193840,"data_compressed_allocated":1043136512,"data_compressed_original":16690184192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:52.729 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:52.729 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 25881, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:51.193752+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:51.193752+0000', 'last_peered': '2024-09-15T20:30:51.193752+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:51.193752+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3440, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1465, 'num_bytes_recovered': 6144655360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:52.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:53.141 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:53.142 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3440/14716 objects degraded (23.376%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":330,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1939484672,"bytes_avail":384607571968,"bytes_total":386547056640,"degraded_objects":3440,"degraded_total":14716,"degraded_ratio":0.23375917368850233,"recovering_objects_per_sec":59,"recovering_bytes_per_sec":248474597,"recovering_keys_per_sec":0,"num_objects_recovered":711,"num_bytes_recovered":2982150144,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:53.156 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:53.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:53.480 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:53.480 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:53.494 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-15T20:30:51.610658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1894028,"kb_used_data":1671476,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375593332,"statfs":{"total":386547056640,"available":384607571968,"internally_reserved":0,"allocated":1711591424,"data_stored":27363282788,"data_compressed":39958682,"data_compressed_allocated":1710166016,"data_compressed_original":27362656256,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":711,"num_bytes_recovered":2982150144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001831"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25881,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:51.193752+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:51.193752+0000","last_peered":"2024-09-15T20:30:51.193752+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:51.193752+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1705873408,"data_stored":27293851648,"data_compressed":39858062,"data_compressed_allocated":1705865216,"data_compressed_original":27293843456,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640329,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1061952,"kb_used_data":1023216,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93309888,"statfs":{"total":96636764160,"available":95549325312,"internally_reserved":0,"allocated":1047773184,"data_stored":16758824153,"data_compressed":24293550,"data_compressed_allocated":1047416832,"data_compressed_original":16758669312,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1043136512,"data_stored":16690184192,"data_compressed":24193840,"data_compressed_allocated":1043136512,"data_compressed_original":16690184192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:56.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:56.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:56.829 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T20:30:55.611197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1872364,"kb_used_data":1645652,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375614996,"statfs":{"total":386547056640,"available":384629755904,"internally_reserved":0,"allocated":1685147648,"data_stored":26940182372,"data_compressed":39345362,"data_compressed_allocated":1683722240,"data_compressed_original":26939555840,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":1979711488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001791"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25881,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:51.193752+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:51.193752+0000","last_peered":"2024-09-15T20:30:51.193752+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:51.193752+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1676677120,"data_stored":26826711040,"data_compressed":39180902,"data_compressed_allocated":1676668928,"data_compressed_original":26826702848,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640329,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1040288,"kb_used_data":997392,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93331552,"statfs":{"total":96636764160,"available":95571509248,"internally_reserved":0,"allocated":1021329408,"data_stored":16335723737,"data_compressed":23680230,"data_compressed_allocated":1020973056,"data_compressed_original":16335568896,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1013940224,"data_stored":16223043584,"data_compressed":23516680,"data_compressed_allocated":1013940224,"data_compressed_original":16223043584,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:56.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:57.150 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:57.165 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T20:30:55.611197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1872364,"kb_used_data":1645652,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375614996,"statfs":{"total":386547056640,"available":384629755904,"internally_reserved":0,"allocated":1685147648,"data_stored":26940182372,"data_compressed":39345362,"data_compressed_allocated":1683722240,"data_compressed_original":26939555840,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":1979711488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001791"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":25881,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:51.193752+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:51.193752+0000","last_peered":"2024-09-15T20:30:51.193752+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:51.193752+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":6144655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1676677120,"data_stored":26826711040,"data_compressed":39180902,"data_compressed_allocated":1676668928,"data_compressed_original":26826702848,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640329,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1040288,"kb_used_data":997392,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93331552,"statfs":{"total":96636764160,"available":95571509248,"internally_reserved":0,"allocated":1021329408,"data_stored":16335723737,"data_compressed":23680230,"data_compressed_allocated":1020973056,"data_compressed_original":16335568896,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1013940224,"data_stored":16223043584,"data_compressed":23516680,"data_compressed_allocated":1013940224,"data_compressed_original":16223043584,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:30:57.165 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:30:57.165 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 25881, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:30:51.193752+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:30:51.193752+0000', 'last_peered': '2024-09-15T20:30:51.193752+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:30:51.193752+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3440, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1465, 'num_bytes_recovered': 6144655360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:30:57.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:30:57.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:57.591 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3440/14716 objects degraded (23.376%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":334,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1917300736,"bytes_avail":384629755904,"bytes_total":386547056640,"degraded_objects":3440,"degraded_total":14716,"degraded_ratio":0.23375917368850233,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":164951338,"recovering_keys_per_sec":0,"num_objects_recovered":472,"num_bytes_recovered":1979711488,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:30:57.605 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:30:57.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:30:57.932 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:30:57.932 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:30:57.946 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-15T20:30:57.611466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":7113539584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1872364,"kb_used_data":1645652,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375614996,"statfs":{"total":386547056640,"available":384629755904,"internally_reserved":0,"allocated":1685147648,"data_stored":26940182372,"data_compressed":39345362,"data_compressed_allocated":1683722240,"data_compressed_original":26939555840,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":2948595712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001706"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":26343,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:56.182838+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:56.182838+0000","last_peered":"2024-09-15T20:30:56.182838+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:56.182838+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":7113539584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":7113539584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1676677120,"data_stored":26826711040,"data_compressed":39180902,"data_compressed_allocated":1676668928,"data_compressed_original":26826702848,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640330,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1040288,"kb_used_data":997392,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93331552,"statfs":{"total":96636764160,"available":95571509248,"internally_reserved":0,"allocated":1021329408,"data_stored":16335723737,"data_compressed":23680230,"data_compressed_allocated":1020973056,"data_compressed_original":16335568896,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1013940224,"data_stored":16223043584,"data_compressed":23516680,"data_compressed_allocated":1013940224,"data_compressed_original":16223043584,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:00.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:01.274 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:01.289 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-15T20:30:59.611754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":7113539584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1842188,"kb_used_data":1623796,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375645172,"statfs":{"total":386547056640,"available":384660656128,"internally_reserved":0,"allocated":1662767104,"data_stored":26582093668,"data_compressed":38826282,"data_compressed_allocated":1661341696,"data_compressed_original":26581467136,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":1971322880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001715"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":26343,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:30:56.182838+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:30:56.182838+0000","last_peered":"2024-09-15T20:30:56.182838+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:30:56.182838+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":7113539584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":7113539584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1658195968,"data_stored":26531012608,"data_compressed":38752262,"data_compressed_allocated":1658187776,"data_compressed_original":26531004416,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640330,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1010112,"kb_used_data":975536,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93361728,"statfs":{"total":96636764160,"available":95602409472,"internally_reserved":0,"allocated":998948864,"data_stored":15977635033,"data_compressed":23161150,"data_compressed_allocated":998592512,"data_compressed_original":15977480192,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":995459072,"data_stored":15927345152,"data_compressed":23088040,"data_compressed_allocated":995459072,"data_compressed_original":15927345152,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:01.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:01.622 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:01.623 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:01.635 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-15T20:31:01.612080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1842188,"kb_used_data":1623796,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375645172,"statfs":{"total":386547056640,"available":384660656128,"internally_reserved":0,"allocated":1662767104,"data_stored":26582093668,"data_compressed":38826282,"data_compressed_allocated":1661341696,"data_compressed_original":26581467136,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":706,"num_bytes_recovered":2961178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001718"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":26815,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:01.183946+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:01.183946+0000","last_peered":"2024-09-15T20:31:01.183946+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:01.183946+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1658195968,"data_stored":26531012608,"data_compressed":38752262,"data_compressed_allocated":1658187776,"data_compressed_original":26531004416,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1010112,"kb_used_data":975536,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93361728,"statfs":{"total":96636764160,"available":95602409472,"internally_reserved":0,"allocated":998948864,"data_stored":15977635033,"data_compressed":23161150,"data_compressed_allocated":998592512,"data_compressed_original":15977480192,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":995459072,"data_stored":15927345152,"data_compressed":23088040,"data_compressed_allocated":995459072,"data_compressed_original":15927345152,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:01.636 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:01.636 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 26815, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:01.183946+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:01.183946+0000', 'last_peered': '2024-09-15T20:31:01.183946+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:01.183946+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2973, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1932, 'num_bytes_recovered': 8103395328, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:01.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:02.039 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:02.039 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2973/14716 objects degraded (20.203%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":338,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1886400512,"bytes_avail":384660656128,"bytes_total":386547056640,"degraded_objects":2973,"degraded_total":14716,"degraded_ratio":0.20202500679532481,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":246729557,"recovering_keys_per_sec":0,"num_objects_recovered":706,"num_bytes_recovered":2961178624,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:02.054 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:02.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:02.377 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:02.377 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:02.390 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-15T20:31:01.612080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1842188,"kb_used_data":1623796,"kb_used_omap":8,"kb_used_meta":218295,"kb_avail":375645172,"statfs":{"total":386547056640,"available":384660656128,"internally_reserved":0,"allocated":1662767104,"data_stored":26582093668,"data_compressed":38826282,"data_compressed_allocated":1661341696,"data_compressed_original":26581467136,"omap_allocated":9187,"internal_metadata":223534109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":706,"num_bytes_recovered":2961178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001718"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":26815,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:01.183946+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:01.183946+0000","last_peered":"2024-09-15T20:31:01.183946+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:01.183946+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1658195968,"data_stored":26531012608,"data_compressed":38752262,"data_compressed_allocated":1658187776,"data_compressed_original":26531004416,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1010112,"kb_used_data":975536,"kb_used_omap":4,"kb_used_meta":34555,"kb_avail":93361728,"statfs":{"total":96636764160,"available":95602409472,"internally_reserved":0,"allocated":998948864,"data_stored":15977635033,"data_compressed":23161150,"data_compressed_allocated":998592512,"data_compressed_original":15977480192,"omap_allocated":4404,"internal_metadata":35385036},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":995459072,"data_stored":15927345152,"data_compressed":23088040,"data_compressed_allocated":995459072,"data_compressed_original":15927345152,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:05.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:05.717 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:05.717 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:05.730 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-15T20:31:05.612651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1812108,"kb_used_data":1589556,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375675252,"statfs":{"total":386547056640,"available":384691458048,"internally_reserved":0,"allocated":1627705344,"data_stored":26021105508,"data_compressed":38013082,"data_compressed_allocated":1626279936,"data_compressed_original":26020478976,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001714"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":26815,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:01.183946+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:01.183946+0000","last_peered":"2024-09-15T20:31:01.183946+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:01.183946+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1626378240,"data_stored":26021928960,"data_compressed":38014302,"data_compressed_allocated":1626370048,"data_compressed_original":26021920768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980032,"kb_used_data":941296,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93391808,"statfs":{"total":96636764160,"available":95633211392,"internally_reserved":0,"allocated":963887104,"data_stored":15416646873,"data_compressed":22347950,"data_compressed_allocated":963530752,"data_compressed_original":15416492032,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":963641344,"data_stored":15418261504,"data_compressed":22350080,"data_compressed_allocated":963641344,"data_compressed_original":15418261504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:05.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:06.057 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:06.058 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:06.072 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-15T20:31:05.612651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1812108,"kb_used_data":1589556,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375675252,"statfs":{"total":386547056640,"available":384691458048,"internally_reserved":0,"allocated":1627705344,"data_stored":26021105508,"data_compressed":38013082,"data_compressed_allocated":1626279936,"data_compressed_original":26020478976,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001714"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":26815,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:01.183946+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:01.183946+0000","last_peered":"2024-09-15T20:31:01.183946+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:01.183946+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1626378240,"data_stored":26021928960,"data_compressed":38014302,"data_compressed_allocated":1626370048,"data_compressed_original":26021920768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980032,"kb_used_data":941296,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93391808,"statfs":{"total":96636764160,"available":95633211392,"internally_reserved":0,"allocated":963887104,"data_stored":15416646873,"data_compressed":22347950,"data_compressed_allocated":963530752,"data_compressed_original":15416492032,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":963641344,"data_stored":15418261504,"data_compressed":22350080,"data_compressed_allocated":963641344,"data_compressed_original":15418261504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:06.072 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:06.073 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 26815, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:01.183946+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:01.183946+0000', 'last_peered': '2024-09-15T20:31:01.183946+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:01.183946+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2973, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1932, 'num_bytes_recovered': 8103395328, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:06.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:06.474 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:06.474 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2973/14716 objects degraded (20.203%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":343,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1855598592,"bytes_avail":384691458048,"bytes_total":386547056640,"degraded_objects":2973,"degraded_total":14716,"degraded_ratio":0.20202500679532481,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":163205008,"recovering_keys_per_sec":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:06.488 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:06.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:06.812 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:06.826 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-15T20:31:05.612651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1812108,"kb_used_data":1589556,"kb_used_omap":8,"kb_used_meta":222455,"kb_avail":375675252,"statfs":{"total":386547056640,"available":384691458048,"internally_reserved":0,"allocated":1627705344,"data_stored":26021105508,"data_compressed":38013082,"data_compressed_allocated":1626279936,"data_compressed_original":26020478976,"omap_allocated":9187,"internal_metadata":227793949},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001714"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":26815,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:01.183946+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:01.183946+0000","last_peered":"2024-09-15T20:31:01.183946+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:01.183946+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":8103395328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1626378240,"data_stored":26021928960,"data_compressed":38014302,"data_compressed_allocated":1626370048,"data_compressed_original":26021920768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640331,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980032,"kb_used_data":941296,"kb_used_omap":4,"kb_used_meta":38715,"kb_avail":93391808,"statfs":{"total":96636764160,"available":95633211392,"internally_reserved":0,"allocated":963887104,"data_stored":15416646873,"data_compressed":22347950,"data_compressed_allocated":963530752,"data_compressed_original":15416492032,"omap_allocated":4404,"internal_metadata":39644876},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":963641344,"data_stored":15418261504,"data_compressed":22350080,"data_compressed_allocated":963641344,"data_compressed_original":15418261504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:09.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:10.154 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:10.169 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-15T20:31:09.613233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1791500,"kb_used_data":1564788,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375695860,"statfs":{"total":386547056640,"available":384712560640,"internally_reserved":0,"allocated":1602342912,"data_stored":25615306596,"data_compressed":37424842,"data_compressed_allocated":1600917504,"data_compressed_original":25614680064,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001767"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":27277,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:06.175711+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:06.175711+0000","last_peered":"2024-09-15T20:31:06.175711+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:06.175711+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1598590976,"data_stored":25577332736,"data_compressed":37369822,"data_compressed_allocated":1598582784,"data_compressed_original":25577324544,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959424,"kb_used_data":916528,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93412416,"statfs":{"total":96636764160,"available":95654313984,"internally_reserved":0,"allocated":938524672,"data_stored":15010847961,"data_compressed":21759710,"data_compressed_allocated":938168320,"data_compressed_original":15010693120,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":935854080,"data_stored":14973665280,"data_compressed":21705600,"data_compressed_allocated":935854080,"data_compressed_original":14973665280,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:10.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:10.493 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:10.493 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:10.507 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-15T20:31:09.613233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1791500,"kb_used_data":1564788,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375695860,"statfs":{"total":386547056640,"available":384712560640,"internally_reserved":0,"allocated":1602342912,"data_stored":25615306596,"data_compressed":37424842,"data_compressed_allocated":1600917504,"data_compressed_original":25614680064,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001767"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":27277,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:06.175711+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:06.175711+0000","last_peered":"2024-09-15T20:31:06.175711+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:06.175711+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1598590976,"data_stored":25577332736,"data_compressed":37369822,"data_compressed_allocated":1598582784,"data_compressed_original":25577324544,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959424,"kb_used_data":916528,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93412416,"statfs":{"total":96636764160,"available":95654313984,"internally_reserved":0,"allocated":938524672,"data_stored":15010847961,"data_compressed":21759710,"data_compressed_allocated":938168320,"data_compressed_original":15010693120,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":935854080,"data_stored":14973665280,"data_compressed":21705600,"data_compressed_allocated":935854080,"data_compressed_original":14973665280,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:10.508 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:10.508 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 27277, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:06.175711+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:06.175711+0000', 'last_peered': '2024-09-15T20:31:06.175711+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:06.175711+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2742, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2163, 'num_bytes_recovered': 9072279552, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:10.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:10.919 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:10.919 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2742/14716 objects degraded (18.633%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":347,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1834496000,"bytes_avail":384712560640,"bytes_total":386547056640,"degraded_objects":2742,"degraded_total":14716,"degraded_ratio":0.18632780646914923,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":163204289,"recovering_keys_per_sec":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:10.934 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:10.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:11.265 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:11.265 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:11.279 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-15T20:31:09.613233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1791500,"kb_used_data":1564788,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375695860,"statfs":{"total":386547056640,"available":384712560640,"internally_reserved":0,"allocated":1602342912,"data_stored":25615306596,"data_compressed":37424842,"data_compressed_allocated":1600917504,"data_compressed_original":25614680064,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001767"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":27277,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:06.175711+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:06.175711+0000","last_peered":"2024-09-15T20:31:06.175711+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:06.175711+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2163,"num_bytes_recovered":9072279552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1598590976,"data_stored":25577332736,"data_compressed":37369822,"data_compressed_allocated":1598582784,"data_compressed_original":25577324544,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959424,"kb_used_data":916528,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93412416,"statfs":{"total":96636764160,"available":95654313984,"internally_reserved":0,"allocated":938524672,"data_stored":15010847961,"data_compressed":21759710,"data_compressed_allocated":938168320,"data_compressed_original":15010693120,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":935854080,"data_stored":14973665280,"data_compressed":21705600,"data_compressed_allocated":935854080,"data_compressed_original":14973665280,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:14.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:14.628 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-15T20:31:13.613814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1791500,"kb_used_data":1564788,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375695860,"statfs":{"total":386547056640,"available":384712560640,"internally_reserved":0,"allocated":1602342912,"data_stored":25615306596,"data_compressed":37424842,"data_compressed_allocated":1600917504,"data_compressed_original":25614680064,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":1933574144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001733"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":27737,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:11.187826+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:11.187826+0000","last_peered":"2024-09-15T20:31:11.187826+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:11.187826+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1598590976,"data_stored":25577332736,"data_compressed":37369822,"data_compressed_allocated":1598582784,"data_compressed_original":25577324544,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959424,"kb_used_data":916528,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93412416,"statfs":{"total":96636764160,"available":95654313984,"internally_reserved":0,"allocated":938524672,"data_stored":15010847961,"data_compressed":21759710,"data_compressed_allocated":938168320,"data_compressed_original":15010693120,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":935854080,"data_stored":14973665280,"data_compressed":21705600,"data_compressed_allocated":935854080,"data_compressed_original":14973665280,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:14.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:14.960 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:14.960 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:14.973 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-15T20:31:13.613814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1791500,"kb_used_data":1564788,"kb_used_omap":8,"kb_used_meta":226615,"kb_avail":375695860,"statfs":{"total":386547056640,"available":384712560640,"internally_reserved":0,"allocated":1602342912,"data_stored":25615306596,"data_compressed":37424842,"data_compressed_allocated":1600917504,"data_compressed_original":25614680064,"omap_allocated":9187,"internal_metadata":232053789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":1933574144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001733"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":27737,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:11.187826+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:11.187826+0000","last_peered":"2024-09-15T20:31:11.187826+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:11.187826+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1598590976,"data_stored":25577332736,"data_compressed":37369822,"data_compressed_allocated":1598582784,"data_compressed_original":25577324544,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959424,"kb_used_data":916528,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":93412416,"statfs":{"total":96636764160,"available":95654313984,"internally_reserved":0,"allocated":938524672,"data_stored":15010847961,"data_compressed":21759710,"data_compressed_allocated":938168320,"data_compressed_original":15010693120,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":935854080,"data_stored":14973665280,"data_compressed":21705600,"data_compressed_allocated":935854080,"data_compressed_original":14973665280,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:14.974 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:14.974 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 27737, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:11.187826+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:11.187826+0000', 'last_peered': '2024-09-15T20:31:11.187826+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:11.187826+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2512, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2393, 'num_bytes_recovered': 10036969472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:14.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:15.377 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:15.377 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2512/14716 objects degraded (17.070%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1834496000,"bytes_avail":384712560640,"bytes_total":386547056640,"degraded_objects":2512,"degraded_total":14716,"degraded_ratio":0.1706985593911389,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":161107901,"recovering_keys_per_sec":0,"num_objects_recovered":461,"num_bytes_recovered":1933574144,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:15.391 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:15.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:15.714 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:15.714 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:15.728 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-15T20:31:15.614080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1780396,"kb_used_data":1543956,"kb_used_omap":8,"kb_used_meta":236343,"kb_avail":375706964,"statfs":{"total":386547056640,"available":384723931136,"internally_reserved":0,"allocated":1581010944,"data_stored":25273995108,"data_compressed":36930082,"data_compressed_allocated":1579585536,"data_compressed_original":25273368576,"omap_allocated":9187,"internal_metadata":242015261},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":1933574144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001699"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":27737,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:11.187826+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:11.187826+0000","last_peered":"2024-09-15T20:31:11.187826+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:11.187826+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":10036969472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1578930176,"data_stored":25262759936,"data_compressed":36913822,"data_compressed_allocated":1578921984,"data_compressed_original":25262751744,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948320,"kb_used_data":895696,"kb_used_omap":4,"kb_used_meta":52603,"kb_avail":93423520,"statfs":{"total":96636764160,"available":95665684480,"internally_reserved":0,"allocated":917192704,"data_stored":14669536473,"data_compressed":21264950,"data_compressed_allocated":916836352,"data_compressed_original":14669381632,"omap_allocated":4404,"internal_metadata":53866188},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":916193280,"data_stored":14659092480,"data_compressed":21249600,"data_compressed_allocated":916193280,"data_compressed_original":14659092480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:18.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:19.065 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:19.066 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:19.079 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-15T20:31:17.614368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1780396,"kb_used_data":1543956,"kb_used_omap":8,"kb_used_meta":236343,"kb_avail":375706964,"statfs":{"total":386547056640,"available":384723931136,"internally_reserved":0,"allocated":1581010944,"data_stored":25273995108,"data_compressed":36930082,"data_compressed_allocated":1579585536,"data_compressed_original":25273368576,"omap_allocated":9187,"internal_metadata":242015261},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2919235584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001717"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":28207,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:16.183406+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:16.183406+0000","last_peered":"2024-09-15T20:31:16.183406+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:16.183406+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1578930176,"data_stored":25262759936,"data_compressed":36913822,"data_compressed_allocated":1578921984,"data_compressed_original":25262751744,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948320,"kb_used_data":895696,"kb_used_omap":4,"kb_used_meta":52603,"kb_avail":93423520,"statfs":{"total":96636764160,"available":95665684480,"internally_reserved":0,"allocated":917192704,"data_stored":14669536473,"data_compressed":21264950,"data_compressed_allocated":916836352,"data_compressed_original":14669381632,"omap_allocated":4404,"internal_metadata":53866188},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":916193280,"data_stored":14659092480,"data_compressed":21249600,"data_compressed_allocated":916193280,"data_compressed_original":14659092480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:19.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:19.405 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:19.405 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:19.419 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-15T20:31:17.614368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1780396,"kb_used_data":1543956,"kb_used_omap":8,"kb_used_meta":236343,"kb_avail":375706964,"statfs":{"total":386547056640,"available":384723931136,"internally_reserved":0,"allocated":1581010944,"data_stored":25273995108,"data_compressed":36930082,"data_compressed_allocated":1579585536,"data_compressed_original":25273368576,"omap_allocated":9187,"internal_metadata":242015261},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":2919235584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001717"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":28207,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:16.183406+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:16.183406+0000","last_peered":"2024-09-15T20:31:16.183406+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:16.183406+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1578930176,"data_stored":25262759936,"data_compressed":36913822,"data_compressed_allocated":1578921984,"data_compressed_original":25262751744,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":948320,"kb_used_data":895696,"kb_used_omap":4,"kb_used_meta":52603,"kb_avail":93423520,"statfs":{"total":96636764160,"available":95665684480,"internally_reserved":0,"allocated":917192704,"data_stored":14669536473,"data_compressed":21264950,"data_compressed_allocated":916836352,"data_compressed_original":14669381632,"omap_allocated":4404,"internal_metadata":53866188},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":916193280,"data_stored":14659092480,"data_compressed":21249600,"data_compressed_allocated":916193280,"data_compressed_original":14659092480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:19.419 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:19.419 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 28207, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:16.183406+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:16.183406+0000', 'last_peered': '2024-09-15T20:31:16.183406+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:16.183406+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2628, 'num_bytes_recovered': 11022630912, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:19.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:19.844 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:19.844 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2277/14716 objects degraded (15.473%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":356,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1795633152,"bytes_avail":384751423488,"bytes_total":386547056640,"degraded_objects":2277,"degraded_total":14716,"degraded_ratio":0.15472954607230224,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":162506123,"recovering_keys_per_sec":0,"num_objects_recovered":465,"num_bytes_recovered":1950351360,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:19.855 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:19.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:20.185 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:20.185 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:20.198 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-15T20:31:19.614669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1753548,"kb_used_data":1512948,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375733812,"statfs":{"total":386547056640,"available":384751423488,"internally_reserved":0,"allocated":1549258752,"data_stored":24765960036,"data_compressed":36193642,"data_compressed_allocated":1547833344,"data_compressed_original":24765333504,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":1950351360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001709"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":28207,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:16.183406+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:16.183406+0000","last_peered":"2024-09-15T20:31:16.183406+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:16.183406+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2628,"num_bytes_recovered":11022630912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1546031104,"data_stored":24736374784,"data_compressed":36150782,"data_compressed_allocated":1546022912,"data_compressed_original":24736366592,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921472,"kb_used_data":864688,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93450368,"statfs":{"total":96636764160,"available":95693176832,"internally_reserved":0,"allocated":885440512,"data_stored":14161501401,"data_compressed":20528510,"data_compressed_allocated":885084160,"data_compressed_original":14161346560,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":883294208,"data_stored":14132707328,"data_compressed":20486560,"data_compressed_allocated":883294208,"data_compressed_original":14132707328,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:23.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:23.522 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:23.523 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:23.536 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-15T20:31:21.614961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1753548,"kb_used_data":1512948,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375733812,"statfs":{"total":386547056640,"available":384751423488,"internally_reserved":0,"allocated":1549258752,"data_stored":24765960036,"data_compressed":36193642,"data_compressed_allocated":1547833344,"data_compressed_original":24765333504,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":697,"num_bytes_recovered":2923429888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001727"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":28671,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:21.186700+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:21.186700+0000","last_peered":"2024-09-15T20:31:21.186700+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:21.186700+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1546031104,"data_stored":24736374784,"data_compressed":36150782,"data_compressed_allocated":1546022912,"data_compressed_original":24736366592,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921472,"kb_used_data":864688,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93450368,"statfs":{"total":96636764160,"available":95693176832,"internally_reserved":0,"allocated":885440512,"data_stored":14161501401,"data_compressed":20528510,"data_compressed_allocated":885084160,"data_compressed_original":14161346560,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":883294208,"data_stored":14132707328,"data_compressed":20486560,"data_compressed_allocated":883294208,"data_compressed_original":14132707328,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:23.536 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:23.862 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:23.863 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:23.876 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-15T20:31:23.615273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1753548,"kb_used_data":1512948,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375733812,"statfs":{"total":386547056640,"available":384751423488,"internally_reserved":0,"allocated":1549258752,"data_stored":24765960036,"data_compressed":36193642,"data_compressed_allocated":1547833344,"data_compressed_original":24765333504,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001741"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":28671,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:21.186700+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:21.186700+0000","last_peered":"2024-09-15T20:31:21.186700+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:21.186700+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1546031104,"data_stored":24736374784,"data_compressed":36150782,"data_compressed_allocated":1546022912,"data_compressed_original":24736366592,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921472,"kb_used_data":864688,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93450368,"statfs":{"total":96636764160,"available":95693176832,"internally_reserved":0,"allocated":885440512,"data_stored":14161501401,"data_compressed":20528510,"data_compressed_allocated":885084160,"data_compressed_original":14161346560,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":883294208,"data_stored":14132707328,"data_compressed":20486560,"data_compressed_allocated":883294208,"data_compressed_original":14132707328,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:23.877 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:23.877 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 28671, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:21.186700+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:21.186700+0000', 'last_peered': '2024-09-15T20:31:21.186700+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:21.186700+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2860, 'num_bytes_recovered': 11995709440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:23.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:24.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:24.274 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2045/14716 objects degraded (13.896%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":361,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1795633152,"bytes_avail":384751423488,"bytes_total":386547056640,"degraded_objects":2045,"degraded_total":14716,"degraded_ratio":0.13896439249796141,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":163204648,"recovering_keys_per_sec":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:24.288 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:24.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:24.618 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:24.618 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:24.631 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-15T20:31:23.615273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1753548,"kb_used_data":1512948,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375733812,"statfs":{"total":386547056640,"available":384751423488,"internally_reserved":0,"allocated":1549258752,"data_stored":24765960036,"data_compressed":36193642,"data_compressed_allocated":1547833344,"data_compressed_original":24765333504,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":1958739968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001741"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":28671,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:21.186700+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:21.186700+0000","last_peered":"2024-09-15T20:31:21.186700+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:21.186700+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":11995709440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1546031104,"data_stored":24736374784,"data_compressed":36150782,"data_compressed_allocated":1546022912,"data_compressed_original":24736366592,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921472,"kb_used_data":864688,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93450368,"statfs":{"total":96636764160,"available":95693176832,"internally_reserved":0,"allocated":885440512,"data_stored":14161501401,"data_compressed":20528510,"data_compressed_allocated":885084160,"data_compressed_original":14161346560,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":883294208,"data_stored":14132707328,"data_compressed":20486560,"data_compressed_allocated":883294208,"data_compressed_original":14132707328,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:27.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:27.955 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:27.956 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:27.969 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-15T20:31:27.615932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1727660,"kb_used_data":1482900,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375759700,"statfs":{"total":386547056640,"available":384777932800,"internally_reserved":0,"allocated":1518489600,"data_stored":24273653604,"data_compressed":35480002,"data_compressed_allocated":1517064192,"data_compressed_original":24273027072,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":2952790016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001851"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29145,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:26.189884+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:26.189884+0000","last_peered":"2024-09-15T20:31:26.189884+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:26.189884+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1515950080,"data_stored":24255078400,"data_compressed":35453102,"data_compressed_allocated":1515941888,"data_compressed_original":24255070208,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895584,"kb_used_data":834640,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93476256,"statfs":{"total":96636764160,"available":95719686144,"internally_reserved":0,"allocated":854671360,"data_stored":13669194969,"data_compressed":19814870,"data_compressed_allocated":854315008,"data_compressed_original":13669040128,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":853213184,"data_stored":13651410944,"data_compressed":19788880,"data_compressed_allocated":853213184,"data_compressed_original":13651410944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:27.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:28.291 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:28.291 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:28.305 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-15T20:31:27.615932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1727660,"kb_used_data":1482900,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375759700,"statfs":{"total":386547056640,"available":384777932800,"internally_reserved":0,"allocated":1518489600,"data_stored":24273653604,"data_compressed":35480002,"data_compressed_allocated":1517064192,"data_compressed_original":24273027072,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":2952790016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001851"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29145,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:26.189884+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:26.189884+0000","last_peered":"2024-09-15T20:31:26.189884+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:26.189884+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1515950080,"data_stored":24255078400,"data_compressed":35453102,"data_compressed_allocated":1515941888,"data_compressed_original":24255070208,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895584,"kb_used_data":834640,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93476256,"statfs":{"total":96636764160,"available":95719686144,"internally_reserved":0,"allocated":854671360,"data_stored":13669194969,"data_compressed":19814870,"data_compressed_allocated":854315008,"data_compressed_original":13669040128,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":853213184,"data_stored":13651410944,"data_compressed":19788880,"data_compressed_allocated":853213184,"data_compressed_original":13651410944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:28.305 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:28.306 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 29145, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:26.189884+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:26.189884+0000', 'last_peered': '2024-09-15T20:31:26.189884+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:26.189884+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1808, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3097, 'num_bytes_recovered': 12989759488, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:28.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1808/14716 objects degraded (12.286%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":365,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1769123840,"bytes_avail":384777932800,"bytes_total":386547056640,"degraded_objects":1808,"degraded_total":14716,"degraded_ratio":0.12285947268279424,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":246027864,"recovering_keys_per_sec":0,"num_objects_recovered":704,"num_bytes_recovered":2952790016,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:28.722 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:28.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:29.050 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:29.050 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:29.064 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-15T20:31:27.615932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1727660,"kb_used_data":1482900,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375759700,"statfs":{"total":386547056640,"available":384777932800,"internally_reserved":0,"allocated":1518489600,"data_stored":24273653604,"data_compressed":35480002,"data_compressed_allocated":1517064192,"data_compressed_original":24273027072,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-704,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":2952790016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001851"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29145,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:26.189884+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:26.189884+0000","last_peered":"2024-09-15T20:31:26.189884+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:26.189884+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":71,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388366+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388366+0000","last_peered":"2024-09-15T20:30:20.388366+0000","last_clean":"2024-09-15T20:30:20.388366+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388366+0000","last_undegraded":"2024-09-15T20:30:20.388366+0000","last_fullsized":"2024-09-15T20:30:20.388366+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3097,"num_bytes_recovered":12989759488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1515950080,"data_stored":24255078400,"data_compressed":35453102,"data_compressed_allocated":1515941888,"data_compressed_original":24255070208,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895584,"kb_used_data":834640,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93476256,"statfs":{"total":96636764160,"available":95719686144,"internally_reserved":0,"allocated":854671360,"data_stored":13669194969,"data_compressed":19814870,"data_compressed_allocated":854315008,"data_compressed_original":13669040128,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705742,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":853213184,"data_stored":13651410944,"data_compressed":19788880,"data_compressed_allocated":853213184,"data_compressed_original":13651410944,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:32.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:32.379 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:32.380 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:32.393 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-15T20:31:31.616476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1697804,"kb_used_data":1461364,"kb_used_omap":8,"kb_used_meta":236343,"kb_avail":375789556,"statfs":{"total":386547056640,"available":384808505344,"internally_reserved":0,"allocated":1496436736,"data_stored":23920807780,"data_compressed":34968522,"data_compressed_allocated":1495011328,"data_compressed_original":23920181248,"omap_allocated":9187,"internal_metadata":242015261},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":2944401408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001807"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29611,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:31.204626+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:31.204626+0000","last_peered":"2024-09-15T20:31:31.204626+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:31.204626+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1493929984,"data_stored":23902756864,"data_compressed":34942382,"data_compressed_allocated":1493921792,"data_compressed_original":23902748672,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865728,"kb_used_data":813104,"kb_used_omap":4,"kb_used_meta":52603,"kb_avail":93506112,"statfs":{"total":96636764160,"available":95750258688,"internally_reserved":0,"allocated":832618496,"data_stored":13316349145,"data_compressed":19303390,"data_compressed_allocated":832262144,"data_compressed_original":13316194304,"omap_allocated":4404,"internal_metadata":53866188},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":831193088,"data_stored":13299089408,"data_compressed":19278160,"data_compressed_allocated":831193088,"data_compressed_original":13299089408,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:32.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:32.724 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:32.724 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:32.739 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-15T20:31:31.616476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1697804,"kb_used_data":1461364,"kb_used_omap":8,"kb_used_meta":236343,"kb_avail":375789556,"statfs":{"total":386547056640,"available":384808505344,"internally_reserved":0,"allocated":1496436736,"data_stored":23920807780,"data_compressed":34968522,"data_compressed_allocated":1495011328,"data_compressed_original":23920181248,"omap_allocated":9187,"internal_metadata":242015261},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":2944401408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001807"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29611,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:31.204626+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:31.204626+0000","last_peered":"2024-09-15T20:31:31.204626+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:31.204626+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1493929984,"data_stored":23902756864,"data_compressed":34942382,"data_compressed_allocated":1493921792,"data_compressed_original":23902748672,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865728,"kb_used_data":813104,"kb_used_omap":4,"kb_used_meta":52603,"kb_avail":93506112,"statfs":{"total":96636764160,"available":95750258688,"internally_reserved":0,"allocated":832618496,"data_stored":13316349145,"data_compressed":19303390,"data_compressed_allocated":832262144,"data_compressed_original":13316194304,"omap_allocated":4404,"internal_metadata":53866188},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":831193088,"data_stored":13299089408,"data_compressed":19278160,"data_compressed_allocated":831193088,"data_compressed_original":13299089408,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:32.739 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:32.739 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 29611, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:31.204626+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:31.204626+0000', 'last_peered': '2024-09-15T20:31:31.204626+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:31.204626+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1575, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3330, 'num_bytes_recovered': 13967032320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:32.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:33.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:33.142 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1575/14716 objects degraded (10.703%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":370,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1738551296,"bytes_avail":384808505344,"bytes_total":386547056640,"degraded_objects":1575,"degraded_total":14716,"degraded_ratio":0.10702636586028812,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":245329835,"recovering_keys_per_sec":0,"num_objects_recovered":702,"num_bytes_recovered":2944401408,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:33.157 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:33.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:33.478 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:33.479 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:33.492 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-15T20:31:31.616476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1697804,"kb_used_data":1461364,"kb_used_omap":8,"kb_used_meta":236343,"kb_avail":375789556,"statfs":{"total":386547056640,"available":384808505344,"internally_reserved":0,"allocated":1496436736,"data_stored":23920807780,"data_compressed":34968522,"data_compressed_allocated":1495011328,"data_compressed_original":23920181248,"omap_allocated":9187,"internal_metadata":242015261},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":2944401408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001807"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29611,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:31.204626+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:31.204626+0000","last_peered":"2024-09-15T20:31:31.204626+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:31.204626+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1493929984,"data_stored":23902756864,"data_compressed":34942382,"data_compressed_allocated":1493921792,"data_compressed_original":23902748672,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865728,"kb_used_data":813104,"kb_used_omap":4,"kb_used_meta":52603,"kb_avail":93506112,"statfs":{"total":96636764160,"available":95750258688,"internally_reserved":0,"allocated":832618496,"data_stored":13316349145,"data_compressed":19303390,"data_compressed_allocated":832262144,"data_compressed_original":13316194304,"omap_allocated":4404,"internal_metadata":53866188},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":831193088,"data_stored":13299089408,"data_compressed":19278160,"data_compressed_allocated":831193088,"data_compressed_original":13299089408,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:36.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:36.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:36.819 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:36.834 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-15T20:31:35.617063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1672332,"kb_used_data":1431732,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375815028,"statfs":{"total":386547056640,"available":384834588672,"internally_reserved":0,"allocated":1466093568,"data_stored":23435317092,"data_compressed":34264762,"data_compressed_allocated":1464668160,"data_compressed_original":23434690560,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":1971322880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001789"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29611,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:31.204626+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:31.204626+0000","last_peered":"2024-09-15T20:31:31.204626+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:31.204626+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1464733696,"data_stored":23435616256,"data_compressed":34265222,"data_compressed_allocated":1464725504,"data_compressed_original":23435608064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840256,"kb_used_data":783472,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93531584,"statfs":{"total":96636764160,"available":95776342016,"internally_reserved":0,"allocated":802275328,"data_stored":12830858457,"data_compressed":18599630,"data_compressed_allocated":801918976,"data_compressed_original":12830703616,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":801996800,"data_stored":12831948800,"data_compressed":18601000,"data_compressed_allocated":801996800,"data_compressed_original":12831948800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:36.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:37.160 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:37.161 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:37.175 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-15T20:31:35.617063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1672332,"kb_used_data":1431732,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375815028,"statfs":{"total":386547056640,"available":384834588672,"internally_reserved":0,"allocated":1466093568,"data_stored":23435317092,"data_compressed":34264762,"data_compressed_allocated":1464668160,"data_compressed_original":23434690560,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":1971322880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001789"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":29611,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:31.204626+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:31.204626+0000","last_peered":"2024-09-15T20:31:31.204626+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:31.204626+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":13967032320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1464733696,"data_stored":23435616256,"data_compressed":34265222,"data_compressed_allocated":1464725504,"data_compressed_original":23435608064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840256,"kb_used_data":783472,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93531584,"statfs":{"total":96636764160,"available":95776342016,"internally_reserved":0,"allocated":802275328,"data_stored":12830858457,"data_compressed":18599630,"data_compressed_allocated":801918976,"data_compressed_original":12830703616,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":801996800,"data_stored":12831948800,"data_compressed":18601000,"data_compressed_allocated":801996800,"data_compressed_original":12831948800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:37.176 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:37.176 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 29611, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:31.204626+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:31.204626+0000', 'last_peered': '2024-09-15T20:31:31.204626+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:31.204626+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1575, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3330, 'num_bytes_recovered': 13967032320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:37.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:37.578 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:37.578 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1575/14716 objects degraded (10.703%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":374,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1712467968,"bytes_avail":384834588672,"bytes_total":386547056640,"degraded_objects":1575,"degraded_total":14716,"degraded_ratio":0.10702636586028812,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":164252417,"recovering_keys_per_sec":0,"num_objects_recovered":470,"num_bytes_recovered":1971322880,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:37.592 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:37.592 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:37.927 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:37.927 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:37.940 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-15T20:31:37.617349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1672332,"kb_used_data":1431732,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375815028,"statfs":{"total":386547056640,"available":384834588672,"internally_reserved":0,"allocated":1466093568,"data_stored":23435317092,"data_compressed":34264762,"data_compressed_allocated":1464668160,"data_compressed_original":23434690560,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":706,"num_bytes_recovered":2956984344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001699"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":30083,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:36.202406+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:36.202406+0000","last_peered":"2024-09-15T20:31:36.202406+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:36.202406+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1464733696,"data_stored":23435616256,"data_compressed":34265222,"data_compressed_allocated":1464725504,"data_compressed_original":23435608064,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640338,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840256,"kb_used_data":783472,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93531584,"statfs":{"total":96636764160,"available":95776342016,"internally_reserved":0,"allocated":802275328,"data_stored":12830858457,"data_compressed":18599630,"data_compressed_allocated":801918976,"data_compressed_original":12830703616,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":801996800,"data_stored":12831948800,"data_compressed":18601000,"data_compressed_allocated":801996800,"data_compressed_original":12831948800,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:40.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:41.266 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:41.267 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:41.281 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-15T20:31:39.617706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1650380,"kb_used_data":1405620,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375836980,"statfs":{"total":386547056640,"available":384857067520,"internally_reserved":0,"allocated":1439354880,"data_stored":23007498084,"data_compressed":33644602,"data_compressed_allocated":1437929472,"data_compressed_original":23006871552,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":1962934296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001774"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":30083,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:36.202406+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:36.202406+0000","last_peered":"2024-09-15T20:31:36.202406+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:36.202406+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1436651520,"data_stored":22986301440,"data_compressed":33613902,"data_compressed_allocated":1436643328,"data_compressed_original":22986293248,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640338,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":818304,"kb_used_data":757360,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93553536,"statfs":{"total":96636764160,"available":95798820864,"internally_reserved":0,"allocated":775536640,"data_stored":12403039449,"data_compressed":17979470,"data_compressed_allocated":775180288,"data_compressed_original":12402884608,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":773914624,"data_stored":12382633984,"data_compressed":17949680,"data_compressed_allocated":773914624,"data_compressed_original":12382633984,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:41.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:41.605 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:41.619 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-15T20:31:39.617706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1650380,"kb_used_data":1405620,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375836980,"statfs":{"total":386547056640,"available":384857067520,"internally_reserved":0,"allocated":1439354880,"data_stored":23007498084,"data_compressed":33644602,"data_compressed_allocated":1437929472,"data_compressed_original":23006871552,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":1962934296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001774"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":30083,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:36.202406+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:36.202406+0000","last_peered":"2024-09-15T20:31:36.202406+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:36.202406+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":14952693784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1436651520,"data_stored":22986301440,"data_compressed":33613902,"data_compressed_allocated":1436643328,"data_compressed_original":22986293248,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640338,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":818304,"kb_used_data":757360,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93553536,"statfs":{"total":96636764160,"available":95798820864,"internally_reserved":0,"allocated":775536640,"data_stored":12403039449,"data_compressed":17979470,"data_compressed_allocated":775180288,"data_compressed_original":12402884608,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":773914624,"data_stored":12382633984,"data_compressed":17949680,"data_compressed_allocated":773914624,"data_compressed_original":12382633984,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:41.620 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:41.620 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 30083, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:36.202406+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:36.202406+0000', 'last_peered': '2024-09-15T20:31:36.202406+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:36.202406+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3566, 'num_bytes_recovered': 14952693784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:41.620 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:42.021 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:42.021 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1110/14716 objects degraded (7.543%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":378,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1689989120,"bytes_avail":384857067520,"bytes_total":386547056640,"degraded_objects":1110,"degraded_total":14716,"degraded_ratio":0.075428105463441156,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":243582678,"recovering_keys_per_sec":0,"num_objects_recovered":698,"num_bytes_recovered":2923429912,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:42.035 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:42.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:42.358 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:42.358 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:42.372 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-15T20:31:41.617997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1650380,"kb_used_data":1405620,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375836980,"statfs":{"total":386547056640,"available":384857067520,"internally_reserved":0,"allocated":1439354880,"data_stored":23007498084,"data_compressed":33644602,"data_compressed_allocated":1437929472,"data_compressed_original":23006871552,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":698,"num_bytes_recovered":2923429912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001797"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":30541,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:41.199744+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:41.199744+0000","last_peered":"2024-09-15T20:31:41.199744+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:41.199744+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1436651520,"data_stored":22986301440,"data_compressed":33613902,"data_compressed_allocated":1436643328,"data_compressed_original":22986293248,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":818304,"kb_used_data":757360,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93553536,"statfs":{"total":96636764160,"available":95798820864,"internally_reserved":0,"allocated":775536640,"data_stored":12403039449,"data_compressed":17979470,"data_compressed_allocated":775180288,"data_compressed_original":12402884608,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":773914624,"data_stored":12382633984,"data_compressed":17949680,"data_compressed_allocated":773914624,"data_compressed_original":12382633984,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:45.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:45.703 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:45.716 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-15T20:31:45.618585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1613452,"kb_used_data":1381172,"kb_used_omap":8,"kb_used_meta":232183,"kb_avail":375873908,"statfs":{"total":386547056640,"available":384894881792,"internally_reserved":0,"allocated":1414320128,"data_stored":22606942052,"data_compressed":33063962,"data_compressed_allocated":1412894720,"data_compressed_original":22606315520,"omap_allocated":9187,"internal_metadata":237755421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":1946157080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001829"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":30541,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:41.199744+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:41.199744+0000","last_peered":"2024-09-15T20:31:41.199744+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:41.199744+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1411846144,"data_stored":22589415424,"data_compressed":33038582,"data_compressed_allocated":1411837952,"data_compressed_original":22589407232,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":781376,"kb_used_data":732912,"kb_used_omap":4,"kb_used_meta":48443,"kb_avail":93590464,"statfs":{"total":96636764160,"available":95836635136,"internally_reserved":0,"allocated":750501888,"data_stored":12002483417,"data_compressed":17398830,"data_compressed_allocated":750145536,"data_compressed_original":12002328576,"omap_allocated":4404,"internal_metadata":49606348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":749109248,"data_stored":11985747968,"data_compressed":17374360,"data_compressed_allocated":749109248,"data_compressed_original":11985747968,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:45.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:46.049 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:46.063 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-15T20:31:45.618585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1613452,"kb_used_data":1381172,"kb_used_omap":8,"kb_used_meta":232183,"kb_avail":375873908,"statfs":{"total":386547056640,"available":384894881792,"internally_reserved":0,"allocated":1414320128,"data_stored":22606942052,"data_compressed":33063962,"data_compressed_allocated":1412894720,"data_compressed_original":22606315520,"omap_allocated":9187,"internal_metadata":237755421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":1946157080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001829"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":30541,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:41.199744+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:41.199744+0000","last_peered":"2024-09-15T20:31:41.199744+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:41.199744+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1411846144,"data_stored":22589415424,"data_compressed":33038582,"data_compressed_allocated":1411837952,"data_compressed_original":22589407232,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":781376,"kb_used_data":732912,"kb_used_omap":4,"kb_used_meta":48443,"kb_avail":93590464,"statfs":{"total":96636764160,"available":95836635136,"internally_reserved":0,"allocated":750501888,"data_stored":12002483417,"data_compressed":17398830,"data_compressed_allocated":750145536,"data_compressed_original":12002328576,"omap_allocated":4404,"internal_metadata":49606348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":749109248,"data_stored":11985747968,"data_compressed":17374360,"data_compressed_allocated":749109248,"data_compressed_original":11985747968,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:46.064 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:46.064 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 30541, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:41.199744+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:41.199744+0000', 'last_peered': '2024-09-15T20:31:41.199744+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:41.199744+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3795, 'num_bytes_recovered': 15913189400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:46.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:46.477 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1110/14716 objects degraded (7.543%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":383,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1652174848,"bytes_avail":384894881792,"bytes_total":386547056640,"degraded_objects":1110,"degraded_total":14716,"degraded_ratio":0.075428105463441156,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":162155032,"recovering_keys_per_sec":0,"num_objects_recovered":465,"num_bytes_recovered":1946157080,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:46.491 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:46.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:46.810 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:46.810 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:46.824 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-15T20:31:45.618585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1613452,"kb_used_data":1381172,"kb_used_omap":8,"kb_used_meta":232183,"kb_avail":375873908,"statfs":{"total":386547056640,"available":384894881792,"internally_reserved":0,"allocated":1414320128,"data_stored":22606942052,"data_compressed":33063962,"data_compressed_allocated":1412894720,"data_compressed_original":22606315520,"omap_allocated":9187,"internal_metadata":237755421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":1946157080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001829"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":30541,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:41.199744+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:41.199744+0000","last_peered":"2024-09-15T20:31:41.199744+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:41.199744+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3795,"num_bytes_recovered":15913189400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1411846144,"data_stored":22589415424,"data_compressed":33038582,"data_compressed_allocated":1411837952,"data_compressed_original":22589407232,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":781376,"kb_used_data":732912,"kb_used_omap":4,"kb_used_meta":48443,"kb_avail":93590464,"statfs":{"total":96636764160,"available":95836635136,"internally_reserved":0,"allocated":750501888,"data_stored":12002483417,"data_compressed":17398830,"data_compressed_allocated":750145536,"data_compressed_original":12002328576,"omap_allocated":4404,"internal_metadata":49606348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705746,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":749109248,"data_stored":11985747968,"data_compressed":17374360,"data_compressed_allocated":749109248,"data_compressed_original":11985747968,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:49.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:50.148 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:50.148 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:50.162 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-15T20:31:49.619187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1593708,"kb_used_data":1353108,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375893652,"statfs":{"total":386547056640,"available":384915099648,"internally_reserved":0,"allocated":1385582592,"data_stored":22147141476,"data_compressed":32397442,"data_compressed_allocated":1384157184,"data_compressed_original":22146514944,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":1941962752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001837"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31009,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:46.199293+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:46.199293+0000","last_peered":"2024-09-15T20:31:46.199293+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:46.199293+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384321024,"data_stored":22149013504,"data_compressed":32400182,"data_compressed_allocated":1384312832,"data_compressed_original":22149005312,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761632,"kb_used_data":704848,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93610208,"statfs":{"total":96636764160,"available":95856852992,"internally_reserved":0,"allocated":721764352,"data_stored":11542682841,"data_compressed":16732310,"data_compressed_allocated":721408000,"data_compressed_original":11542528000,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":721584128,"data_stored":11545346048,"data_compressed":16735960,"data_compressed_allocated":721584128,"data_compressed_original":11545346048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:50.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:50.489 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:50.489 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:50.502 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-15T20:31:49.619187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1593708,"kb_used_data":1353108,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375893652,"statfs":{"total":386547056640,"available":384915099648,"internally_reserved":0,"allocated":1385582592,"data_stored":22147141476,"data_compressed":32397442,"data_compressed_allocated":1384157184,"data_compressed_original":22146514944,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":1941962752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001837"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31009,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:46.199293+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:46.199293+0000","last_peered":"2024-09-15T20:31:46.199293+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:46.199293+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384321024,"data_stored":22149013504,"data_compressed":32400182,"data_compressed_allocated":1384312832,"data_compressed_original":22149005312,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761632,"kb_used_data":704848,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93610208,"statfs":{"total":96636764160,"available":95856852992,"internally_reserved":0,"allocated":721764352,"data_stored":11542682841,"data_compressed":16732310,"data_compressed_allocated":721408000,"data_compressed_original":11542528000,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":721584128,"data_stored":11545346048,"data_compressed":16735960,"data_compressed_allocated":721584128,"data_compressed_original":11545346048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:50.502 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:50.502 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 31009, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:46.199293+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:46.199293+0000', 'last_peered': '2024-09-15T20:31:46.199293+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:46.199293+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 876, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4029, 'num_bytes_recovered': 16894656536, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:50.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:50.906 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:50.907 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 876/14716 objects degraded (5.953%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":387,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+clean","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1631956992,"bytes_avail":384915099648,"bytes_total":386547056640,"degraded_objects":876,"degraded_total":14716,"degraded_ratio":0.059527045392769776,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":161805447,"recovering_keys_per_sec":0,"num_objects_recovered":463,"num_bytes_recovered":1941962752,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:50.920 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:50.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:51.257 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-15T20:31:49.619187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1593708,"kb_used_data":1353108,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375893652,"statfs":{"total":386547056640,"available":384915099648,"internally_reserved":0,"allocated":1385582592,"data_stored":22147141476,"data_compressed":32397442,"data_compressed_allocated":1384157184,"data_compressed_original":22146514944,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":1941962752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001837"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31009,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:46.199293+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:46.199293+0000","last_peered":"2024-09-15T20:31:46.199293+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:46.199293+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4029,"num_bytes_recovered":16894656536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384321024,"data_stored":22149013504,"data_compressed":32400182,"data_compressed_allocated":1384312832,"data_compressed_original":22149005312,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761632,"kb_used_data":704848,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93610208,"statfs":{"total":96636764160,"available":95856852992,"internally_reserved":0,"allocated":721764352,"data_stored":11542682841,"data_compressed":16732310,"data_compressed_allocated":721408000,"data_compressed_original":11542528000,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705747,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":721584128,"data_stored":11545346048,"data_compressed":16735960,"data_compressed_allocated":721584128,"data_compressed_original":11545346048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:54.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:54.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:54.590 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:54.603 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-15T20:31:53.619779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1593708,"kb_used_data":1353108,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375893652,"statfs":{"total":386547056640,"available":384915099648,"internally_reserved":0,"allocated":1385582592,"data_stored":22147141476,"data_compressed":32397442,"data_compressed_allocated":1384157184,"data_compressed_original":22146514944,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":1946157056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001781"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31470,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:51.203450+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:51.203450+0000","last_peered":"2024-09-15T20:31:51.203450+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:51.203450+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384321024,"data_stored":22149013504,"data_compressed":32400182,"data_compressed_allocated":1384312832,"data_compressed_original":22149005312,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640341,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761632,"kb_used_data":704848,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93610208,"statfs":{"total":96636764160,"available":95856852992,"internally_reserved":0,"allocated":721764352,"data_stored":11542682841,"data_compressed":16732310,"data_compressed_allocated":721408000,"data_compressed_original":11542528000,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":721584128,"data_stored":11545346048,"data_compressed":16735960,"data_compressed_allocated":721584128,"data_compressed_original":11545346048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:54.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:54.937 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:54.937 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:54.950 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-15T20:31:53.619779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1593708,"kb_used_data":1353108,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375893652,"statfs":{"total":386547056640,"available":384915099648,"internally_reserved":0,"allocated":1385582592,"data_stored":22147141476,"data_compressed":32397442,"data_compressed_allocated":1384157184,"data_compressed_original":22146514944,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":1946157056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001781"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31470,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:51.203450+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:51.203450+0000","last_peered":"2024-09-15T20:31:51.203450+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:51.203450+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1384321024,"data_stored":22149013504,"data_compressed":32400182,"data_compressed_allocated":1384312832,"data_compressed_original":22149005312,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640341,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761632,"kb_used_data":704848,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93610208,"statfs":{"total":96636764160,"available":95856852992,"internally_reserved":0,"allocated":721764352,"data_stored":11542682841,"data_compressed":16732310,"data_compressed_allocated":721408000,"data_compressed_original":11542528000,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":721584128,"data_stored":11545346048,"data_compressed":16735960,"data_compressed_allocated":721584128,"data_compressed_original":11545346048,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:54.950 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:54.950 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 31470, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:51.203450+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:51.203450+0000', 'last_peered': '2024-09-15T20:31:51.203450+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:51.203450+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 645, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4259, 'num_bytes_recovered': 17859346456, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:54.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:55.359 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:55.359 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 645/14716 objects degraded (4.383%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":392,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1631956992,"bytes_avail":384915099648,"bytes_total":386547056640,"degraded_objects":645,"degraded_total":14716,"degraded_ratio":0.043829845066594181,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":162155682,"recovering_keys_per_sec":0,"num_objects_recovered":464,"num_bytes_recovered":1946157056,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:55.373 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:55.374 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:55.703 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:55.703 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:55.717 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-15T20:31:55.620051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1579148,"kb_used_data":1334388,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375908212,"statfs":{"total":386547056640,"available":384930009088,"internally_reserved":0,"allocated":1366413312,"data_stored":21840432996,"data_compressed":31952842,"data_compressed_allocated":1364987904,"data_compressed_original":21839806464,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":1946157056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001737"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31470,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:51.203450+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:51.203450+0000","last_peered":"2024-09-15T20:31:51.203450+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:51.203450+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":17859346456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1364234240,"data_stored":21827624960,"data_compressed":31934302,"data_compressed_allocated":1364226048,"data_compressed_original":21827616768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640341,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747072,"kb_used_data":686128,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93624768,"statfs":{"total":96636764160,"available":95871762432,"internally_reserved":0,"allocated":702595072,"data_stored":11235974361,"data_compressed":16287710,"data_compressed_allocated":702238720,"data_compressed_original":11235819520,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":701497344,"data_stored":11223957504,"data_compressed":16270080,"data_compressed_allocated":701497344,"data_compressed_original":11223957504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:58.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:59.051 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:59.064 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-15T20:31:57.620324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1579148,"kb_used_data":1334388,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375908212,"statfs":{"total":386547056640,"available":384930009088,"internally_reserved":0,"allocated":1366413312,"data_stored":21840432996,"data_compressed":31952842,"data_compressed_allocated":1364987904,"data_compressed_original":21839806464,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":2944401408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001739"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31946,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:56.197042+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:56.197042+0000","last_peered":"2024-09-15T20:31:56.197042+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:56.197042+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1364234240,"data_stored":21827624960,"data_compressed":31934302,"data_compressed_allocated":1364226048,"data_compressed_original":21827616768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640342,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747072,"kb_used_data":686128,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93624768,"statfs":{"total":96636764160,"available":95871762432,"internally_reserved":0,"allocated":702595072,"data_stored":11235974361,"data_compressed":16287710,"data_compressed_allocated":702238720,"data_compressed_original":11235819520,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":701497344,"data_stored":11223957504,"data_compressed":16270080,"data_compressed_allocated":701497344,"data_compressed_original":11223957504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:59.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:59.396 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:59.397 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:31:59.410 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-15T20:31:57.620324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1579148,"kb_used_data":1334388,"kb_used_omap":8,"kb_used_meta":244663,"kb_avail":375908212,"statfs":{"total":386547056640,"available":384930009088,"internally_reserved":0,"allocated":1366413312,"data_stored":21840432996,"data_compressed":31952842,"data_compressed_allocated":1364987904,"data_compressed_original":21839806464,"omap_allocated":9187,"internal_metadata":250534941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":2944401408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001739"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31946,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:56.197042+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:56.197042+0000","last_peered":"2024-09-15T20:31:56.197042+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:56.197042+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1364234240,"data_stored":21827624960,"data_compressed":31934302,"data_compressed_allocated":1364226048,"data_compressed_original":21827616768,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640342,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747072,"kb_used_data":686128,"kb_used_omap":4,"kb_used_meta":60923,"kb_avail":93624768,"statfs":{"total":96636764160,"available":95871762432,"internally_reserved":0,"allocated":702595072,"data_stored":11235974361,"data_compressed":16287710,"data_compressed_allocated":702238720,"data_compressed_original":11235819520,"omap_allocated":4404,"internal_metadata":62385868},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705748,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":701497344,"data_stored":11223957504,"data_compressed":16270080,"data_compressed_allocated":701497344,"data_compressed_original":11223957504,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:59.410 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:31:59.410 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 31946, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:31:56.197042+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:31:56.197042+0000', 'last_peered': '2024-09-15T20:31:56.197042+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:31:56.197042+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 408, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4497, 'num_bytes_recovered': 18857590808, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:31:59.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:31:59.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:31:59.827 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 408/14716 objects degraded (2.772%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":396,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1617047552,"bytes_avail":384930009088,"bytes_total":386547056640,"degraded_objects":408,"degraded_total":14716,"degraded_ratio":0.027724925251427019,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":245331230,"recovering_keys_per_sec":0,"num_objects_recovered":702,"num_bytes_recovered":2944401408,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:31:59.840 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:31:59.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:00.167 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:00.181 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-15T20:31:59.620607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1539500,"kb_used_data":1307220,"kb_used_omap":8,"kb_used_meta":232183,"kb_avail":375947860,"statfs":{"total":386547056640,"available":384970608640,"internally_reserved":0,"allocated":1338593280,"data_stored":21395312484,"data_compressed":31307602,"data_compressed_allocated":1337167872,"data_compressed_original":21394685952,"omap_allocated":9187,"internal_metadata":237755421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":1962934272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":31946,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:31:56.197042+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:31:56.197042+0000","last_peered":"2024-09-15T20:31:56.197042+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:31:56.197042+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":18857590808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1336872960,"data_stored":21389844480,"data_compressed":31299702,"data_compressed_allocated":1336864768,"data_compressed_original":21389836288,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640342,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707424,"kb_used_data":658960,"kb_used_omap":4,"kb_used_meta":48443,"kb_avail":93664416,"statfs":{"total":96636764160,"available":95912361984,"internally_reserved":0,"allocated":674775040,"data_stored":10790853849,"data_compressed":15642470,"data_compressed_allocated":674418688,"data_compressed_original":10790699008,"omap_allocated":4404,"internal_metadata":49606348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":674136064,"data_stored":10786177024,"data_compressed":15635480,"data_compressed_allocated":674136064,"data_compressed_original":10786177024,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:03.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:03.501 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:03.501 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:03.515 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-15T20:32:01.620981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1539500,"kb_used_data":1307220,"kb_used_omap":8,"kb_used_meta":232183,"kb_avail":375947860,"statfs":{"total":386547056640,"available":384970608640,"internally_reserved":0,"allocated":1338593280,"data_stored":21395312484,"data_compressed":31307602,"data_compressed_allocated":1337167872,"data_compressed_original":21394685952,"omap_allocated":9187,"internal_metadata":237755421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":701,"num_bytes_recovered":2940207104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001793"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32412,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:32:01.208363+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:32:01.208363+0000","last_peered":"2024-09-15T20:32:01.208363+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:32:01.208363+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1336872960,"data_stored":21389844480,"data_compressed":31299702,"data_compressed_allocated":1336864768,"data_compressed_original":21389836288,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640343,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707424,"kb_used_data":658960,"kb_used_omap":4,"kb_used_meta":48443,"kb_avail":93664416,"statfs":{"total":96636764160,"available":95912361984,"internally_reserved":0,"allocated":674775040,"data_stored":10790853849,"data_compressed":15642470,"data_compressed_allocated":674418688,"data_compressed_original":10790699008,"omap_allocated":4404,"internal_metadata":49606348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":674136064,"data_stored":10786177024,"data_compressed":15635480,"data_compressed_allocated":674136064,"data_compressed_original":10786177024,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:03.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:03.839 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:03.840 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:03.853 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-15T20:32:03.621296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1539500,"kb_used_data":1307220,"kb_used_omap":8,"kb_used_meta":232183,"kb_avail":375947860,"statfs":{"total":386547056640,"available":384970608640,"internally_reserved":0,"allocated":1338593280,"data_stored":21395312484,"data_compressed":31307602,"data_compressed_allocated":1337167872,"data_compressed_original":21394685952,"omap_allocated":9187,"internal_metadata":237755421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":1975517184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001827"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32412,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:32:01.208363+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:32:01.208363+0000","last_peered":"2024-09-15T20:32:01.208363+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:32:01.208363+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1336872960,"data_stored":21389844480,"data_compressed":31299702,"data_compressed_allocated":1336864768,"data_compressed_original":21389836288,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640343,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707424,"kb_used_data":658960,"kb_used_omap":4,"kb_used_meta":48443,"kb_avail":93664416,"statfs":{"total":96636764160,"available":95912361984,"internally_reserved":0,"allocated":674775040,"data_stored":10790853849,"data_compressed":15642470,"data_compressed_allocated":674418688,"data_compressed_original":10790699008,"omap_allocated":4404,"internal_metadata":49606348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":674136064,"data_stored":10786177024,"data_compressed":15635480,"data_compressed_allocated":674136064,"data_compressed_original":10786177024,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:03.854 INFO:tasks.backfill_toofull.ceph_manager:PG 2.0 is not active+clean 2024-09-15T20:32:03.854 INFO:tasks.backfill_toofull.ceph_manager:{'pgid': '2.0', 'version': "23'22860", 'reported_seq': 32412, 'reported_epoch': 42, 'state': 'active+undersized+degraded+remapped+backfilling', 'last_fresh': '2024-09-15T20:32:01.208363+0000', 'last_change': '2024-09-15T20:30:20.390054+0000', 'last_active': '2024-09-15T20:32:01.208363+0000', 'last_peered': '2024-09-15T20:32:01.208363+0000', 'last_clean': '2024-09-15T20:27:57.822721+0000', 'last_became_active': '2024-09-15T20:30:20.389188+0000', 'last_became_peered': '2024-09-15T20:30:20.389188+0000', 'last_unstale': '2024-09-15T20:32:01.208363+0000', 'last_undegraded': '2024-09-15T20:30:20.386297+0000', 'last_fullsized': '2024-09-15T20:30:20.386074+0000', 'mapping_epoch': 41, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 175, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4730, 'num_bytes_recovered': 19834863640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 4904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:32:03.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T20:32:04.255 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:04.255 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 175/14716 objects degraded (1.189%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":401,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726432218,"num_in_osds":4,"osd_in_since":1726431924,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1}],"num_pgs":2,"num_pools":2,"num_objects":4906,"data_bytes":20565000728,"bytes_used":1576448000,"bytes_avail":384970608640,"bytes_total":386547056640,"degraded_objects":175,"degraded_total":14716,"degraded_ratio":0.011891818428920902,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":164601360,"recovering_keys_per_sec":0,"num_objects_recovered":471,"num_bytes_recovered":1975517184,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T20:25:23:126945+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T20:30:29.607397+0000","services":{}},"progress_events":{}} 2024-09-15T20:32:04.269 INFO:tasks.backfill_toofull.ceph_manager:making progress, resetting timeout 2024-09-15T20:32:04.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:04.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:04.605 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-15T20:32:03.621296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1539500,"kb_used_data":1307220,"kb_used_omap":8,"kb_used_meta":232183,"kb_avail":375947860,"statfs":{"total":386547056640,"available":384970608640,"internally_reserved":0,"allocated":1338593280,"data_stored":21395312484,"data_compressed":31307602,"data_compressed_allocated":1337167872,"data_compressed_original":21394685952,"omap_allocated":9187,"internal_metadata":237755421},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":1975517184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001827"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32412,"reported_epoch":42,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-15T20:32:01.208363+0000","last_change":"2024-09-15T20:30:20.390054+0000","last_active":"2024-09-15T20:32:01.208363+0000","last_peered":"2024-09-15T20:32:01.208363+0000","last_clean":"2024-09-15T20:27:57.822721+0000","last_became_active":"2024-09-15T20:30:20.389188+0000","last_became_peered":"2024-09-15T20:30:20.389188+0000","last_unstale":"2024-09-15T20:32:01.208363+0000","last_undegraded":"2024-09-15T20:30:20.386297+0000","last_fullsized":"2024-09-15T20:30:20.386074+0000","mapping_epoch":41,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2147483647,2],"avail_no_missing":["3(0)","2(2)"],"object_location_counts":[{"shards":"2(2),3(0)","objects":4904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4730,"num_bytes_recovered":19834863640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1336872960,"data_stored":21389844480,"data_compressed":31299702,"data_compressed_allocated":1336864768,"data_compressed_original":21389836288,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640343,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709108,"kb_used_data":627932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662732,"statfs":{"total":96636764160,"available":95910637568,"internally_reserved":0,"allocated":643002368,"data_stored":10282494681,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707424,"kb_used_data":658960,"kb_used_omap":4,"kb_used_meta":48443,"kb_avail":93664416,"statfs":{"total":96636764160,"available":95912361984,"internally_reserved":0,"allocated":674775040,"data_stored":10790853849,"data_compressed":15642470,"data_compressed_allocated":674418688,"data_compressed_original":10790699008,"omap_allocated":4404,"internal_metadata":49606348},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":674136064,"data_stored":10786177024,"data_compressed":15635480,"data_compressed_allocated":674136064,"data_compressed_original":10786177024,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:07.606 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:07.899 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:07.899 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:07.911 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-15T20:32:07.622045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1519312,"kb_used_data":1278716,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375968048,"statfs":{"total":386547056640,"available":384991281152,"internally_reserved":0,"allocated":1309405184,"data_stored":20928175898,"data_compressed":30630442,"data_compressed_allocated":1307971584,"data_compressed_original":20927545344,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":1707081728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001438"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32774,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:05.912180+0000","last_change":"2024-09-15T20:32:05.912076+0000","last_active":"2024-09-15T20:32:05.912180+0000","last_peered":"2024-09-15T20:32:05.912180+0000","last_clean":"2024-09-15T20:32:05.912180+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:05.912180+0000","last_undegraded":"2024-09-15T20:32:05.912180+0000","last_fullsized":"2024-09-15T20:32:05.912180+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:09:11.904454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1308397568,"data_stored":20934238208,"data_compressed":30639262,"data_compressed_allocated":1308389376,"data_compressed_original":20934230016,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709112,"kb_used_data":627940,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662728,"statfs":{"total":96636764160,"available":95910633472,"internally_reserved":0,"allocated":643010560,"data_stored":10282498703,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687232,"kb_used_data":630448,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93684608,"statfs":{"total":96636764160,"available":95933038592,"internally_reserved":0,"allocated":645578752,"data_stored":10323713241,"data_compressed":14965310,"data_compressed_allocated":645222400,"data_compressed_original":10323558400,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645660672,"data_stored":10330570752,"data_compressed":14975040,"data_compressed_allocated":645660672,"data_compressed_original":10330570752,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:07.911 INFO:tasks.backfill_toofull.ceph_manager:clean! 2024-09-15T20:32:07.912 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:08.196 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:08.196 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:08.208 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-15T20:32:07.622045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1519312,"kb_used_data":1278716,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375968048,"statfs":{"total":386547056640,"available":384991281152,"internally_reserved":0,"allocated":1309405184,"data_stored":20928175898,"data_compressed":30630442,"data_compressed_allocated":1307971584,"data_compressed_original":20927545344,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":1707081728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001438"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32774,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:05.912180+0000","last_change":"2024-09-15T20:32:05.912076+0000","last_active":"2024-09-15T20:32:05.912180+0000","last_peered":"2024-09-15T20:32:05.912180+0000","last_clean":"2024-09-15T20:32:05.912180+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:05.912180+0000","last_undegraded":"2024-09-15T20:32:05.912180+0000","last_fullsized":"2024-09-15T20:32:05.912180+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:09:11.904454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1308397568,"data_stored":20934238208,"data_compressed":30639262,"data_compressed_allocated":1308389376,"data_compressed_original":20934230016,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709112,"kb_used_data":627940,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662728,"statfs":{"total":96636764160,"available":95910633472,"internally_reserved":0,"allocated":643010560,"data_stored":10282498703,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687232,"kb_used_data":630448,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93684608,"statfs":{"total":96636764160,"available":95933038592,"internally_reserved":0,"allocated":645578752,"data_stored":10323713241,"data_compressed":14965310,"data_compressed_allocated":645222400,"data_compressed_original":10323558400,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645660672,"data_stored":10330570752,"data_compressed":14975040,"data_compressed_allocated":645660672,"data_compressed_original":10330570752,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:08.209 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'22860", 'reported_seq': 32774, 'reported_epoch': 44, 'state': 'active+clean', 'last_fresh': '2024-09-15T20:32:05.912180+0000', 'last_change': '2024-09-15T20:32:05.912076+0000', 'last_active': '2024-09-15T20:32:05.912180+0000', 'last_peered': '2024-09-15T20:32:05.912180+0000', 'last_clean': '2024-09-15T20:32:05.912180+0000', 'last_became_active': '2024-09-15T20:32:05.911603+0000', 'last_became_peered': '2024-09-15T20:32:05.911603+0000', 'last_unstale': '2024-09-15T20:32:05.912180+0000', 'last_undegraded': '2024-09-15T20:32:05.912180+0000', 'last_fullsized': '2024-09-15T20:32:05.912180+0000', 'mapping_epoch': 43, 'log_start': "23'22833", 'ondisk_log_start': "23'22833", 'created': 15, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'last_clean_scrub_stamp': '2024-09-15T20:25:46.631534+0000', 'objects_scrubbed': 0, 'log_size': 27, 'log_dups_size': 3000, 'ondisk_log_size': 27, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T00:09:11.904454+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20564672536, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 22860, 'num_write_kb': 56852482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4904, 'num_bytes_recovered': 20564672536, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T20:32:08.209 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T20:32:08.219 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T20:32:08.219 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:08.499 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:08.499 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:08.511 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-15T20:32:07.622045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1519312,"kb_used_data":1278716,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375968048,"statfs":{"total":386547056640,"available":384991281152,"internally_reserved":0,"allocated":1309405184,"data_stored":20928175898,"data_compressed":30630442,"data_compressed_allocated":1307971584,"data_compressed_original":20927545344,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":1707081728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001438"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32774,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:05.912180+0000","last_change":"2024-09-15T20:32:05.912076+0000","last_active":"2024-09-15T20:32:05.912180+0000","last_peered":"2024-09-15T20:32:05.912180+0000","last_clean":"2024-09-15T20:32:05.912180+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:05.912180+0000","last_undegraded":"2024-09-15T20:32:05.912180+0000","last_fullsized":"2024-09-15T20:32:05.912180+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:09:11.904454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1308397568,"data_stored":20934238208,"data_compressed":30639262,"data_compressed_allocated":1308389376,"data_compressed_original":20934230016,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709112,"kb_used_data":627940,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662728,"statfs":{"total":96636764160,"available":95910633472,"internally_reserved":0,"allocated":643010560,"data_stored":10282498703,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687232,"kb_used_data":630448,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93684608,"statfs":{"total":96636764160,"available":95933038592,"internally_reserved":0,"allocated":645578752,"data_stored":10323713241,"data_compressed":14965310,"data_compressed_allocated":645222400,"data_compressed_original":10323558400,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645660672,"data_stored":10330570752,"data_compressed":14975040,"data_compressed_allocated":645660672,"data_compressed_original":10330570752,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:08.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:08.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:08.791 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:08.803 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-15T20:32:07.622045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1519312,"kb_used_data":1278716,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375968048,"statfs":{"total":386547056640,"available":384991281152,"internally_reserved":0,"allocated":1309405184,"data_stored":20928175898,"data_compressed":30630442,"data_compressed_allocated":1307971584,"data_compressed_original":20927545344,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":1707081728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001438"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32774,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:05.912180+0000","last_change":"2024-09-15T20:32:05.912076+0000","last_active":"2024-09-15T20:32:05.912180+0000","last_peered":"2024-09-15T20:32:05.912180+0000","last_clean":"2024-09-15T20:32:05.912180+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:05.912180+0000","last_undegraded":"2024-09-15T20:32:05.912180+0000","last_fullsized":"2024-09-15T20:32:05.912180+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:09:11.904454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1308397568,"data_stored":20934238208,"data_compressed":30639262,"data_compressed_allocated":1308389376,"data_compressed_original":20934230016,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709112,"kb_used_data":627940,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662728,"statfs":{"total":96636764160,"available":95910633472,"internally_reserved":0,"allocated":643010560,"data_stored":10282498703,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687232,"kb_used_data":630448,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93684608,"statfs":{"total":96636764160,"available":95933038592,"internally_reserved":0,"allocated":645578752,"data_stored":10323713241,"data_compressed":14965310,"data_compressed_allocated":645222400,"data_compressed_original":10323558400,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645660672,"data_stored":10330570752,"data_compressed":14975040,"data_compressed_allocated":645660672,"data_compressed_original":10330570752,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:08.804 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T20:32:08.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:09.086 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:09.087 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:09.099 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-15T20:32:07.622045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1519312,"kb_used_data":1278716,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375968048,"statfs":{"total":386547056640,"available":384991281152,"internally_reserved":0,"allocated":1309405184,"data_stored":20928175898,"data_compressed":30630442,"data_compressed_allocated":1307971584,"data_compressed_original":20927545344,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":1707081728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001438"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32774,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:05.912180+0000","last_change":"2024-09-15T20:32:05.912076+0000","last_active":"2024-09-15T20:32:05.912180+0000","last_peered":"2024-09-15T20:32:05.912180+0000","last_clean":"2024-09-15T20:32:05.912180+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:05.912180+0000","last_undegraded":"2024-09-15T20:32:05.912180+0000","last_fullsized":"2024-09-15T20:32:05.912180+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:09:11.904454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":72,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T20:30:20.388428+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:30:20.388428+0000","last_peered":"2024-09-15T20:30:20.388428+0000","last_clean":"2024-09-15T20:30:20.388428+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:30:20.388428+0000","last_undegraded":"2024-09-15T20:30:20.388428+0000","last_fullsized":"2024-09-15T20:30:20.388428+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1308397568,"data_stored":20934238208,"data_compressed":30639262,"data_compressed_allocated":1308389376,"data_compressed_original":20934230016,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709112,"kb_used_data":627940,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662728,"statfs":{"total":96636764160,"available":95910633472,"internally_reserved":0,"allocated":643010560,"data_stored":10282498703,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95568,"kb_used_data":19960,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276272,"statfs":{"total":96636764160,"available":96538902528,"internally_reserved":0,"allocated":20439040,"data_stored":321481433,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687232,"kb_used_data":630448,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93684608,"statfs":{"total":96636764160,"available":95933038592,"internally_reserved":0,"allocated":645578752,"data_stored":10323713241,"data_compressed":14965310,"data_compressed_allocated":645222400,"data_compressed_original":10323558400,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705750,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":376832,"data_stored":482521,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645660672,"data_stored":10330570752,"data_compressed":14975040,"data_compressed_allocated":645660672,"data_compressed_original":10330570752,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:09.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:32:09.383 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:09.384 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":44,"fsid":"54dc298d-7b27-4c9d-9204-d7c9a8f9d562","created":"2024-09-15T20:25:23.129567+0000","modified":"2024-09-15T20:32:05.896372+0000","last_up_change":"2024-09-15T20:30:18.355113+0000","last_in_change":"2024-09-15T20:25:24.460612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.21600000560283661,"nearfull_ratio":0.20399999618530273,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:25:33.574537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2024-09-15T20:25:45.636932+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_toofull","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"a52846f6-f66b-4721-b2f4-c44999f2c728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6809","nonce":4274430252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6811","nonce":4274430252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6815","nonce":4274430252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":4274430252},{"type":"v1","addr":"172.21.15.129:6813","nonce":4274430252}]},"public_addr":"172.21.15.129:6809/4274430252","cluster_addr":"172.21.15.129:6811/4274430252","heartbeat_back_addr":"172.21.15.129:6815/4274430252","heartbeat_front_addr":"172.21.15.129:6813/4274430252","state":["exists","up"]},{"osd":1,"uuid":"e3b9918e-a312-4729-bc3d-392d02ead18c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":28,"last_clean_end":39,"up_from":40,"up_thru":40,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":1328262137},{"type":"v1","addr":"172.21.15.129:6801","nonce":1328262137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1329262137},{"type":"v1","addr":"172.21.15.129:6835","nonce":1329262137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":1328262137},{"type":"v1","addr":"172.21.15.129:6807","nonce":1328262137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":1328262137},{"type":"v1","addr":"172.21.15.129:6805","nonce":1328262137}]},"public_addr":"172.21.15.129:6801/1328262137","cluster_addr":"172.21.15.129:6835/1329262137","heartbeat_back_addr":"172.21.15.129:6807/1328262137","heartbeat_front_addr":"172.21.15.129:6805/1328262137","state":["exists","up"]},{"osd":2,"uuid":"a7c6897c-1a1d-4652-94b3-764fe468eeae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6825","nonce":3344078257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6827","nonce":3344078257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6831","nonce":3344078257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3344078257},{"type":"v1","addr":"172.21.15.129:6829","nonce":3344078257}]},"public_addr":"172.21.15.129:6825/3344078257","cluster_addr":"172.21.15.129:6827/3344078257","heartbeat_back_addr":"172.21.15.129:6831/3344078257","heartbeat_front_addr":"172.21.15.129:6829/3344078257","state":["exists","up"]},{"osd":3,"uuid":"4858f290-1285-44ae-8797-0c63c6dcbef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6817","nonce":435893452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6819","nonce":435893452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6823","nonce":435893452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":435893452},{"type":"v1","addr":"172.21.15.129:6821","nonce":435893452}]},"public_addr":"172.21.15.129:6817/435893452","cluster_addr":"172.21.15.129:6819/435893452","heartbeat_back_addr":"172.21.15.129:6823/435893452","heartbeat_front_addr":"172.21.15.129:6821/435893452","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.577650+0000","dead_epoch":0},{"osd":1,"down_stamp":"2024-09-15T20:30:13.334557+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:30.816456+0000","dead_epoch":38},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.038929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:25:31.251155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_toofull":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:32:10.397 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T20:32:10.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:32:10.523 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-15T20:32:10.523 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:32:10.523 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-15T20:32:10.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T20:32:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T20:32:10.831 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T20:32:10.832 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:32:10.953 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-15T20:32:10.953 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:32:10.953 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-15T20:32:10.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T20:32:11.248 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T20:32:11.261 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T20:32:11.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:32:11.382 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-15T20:32:11.382 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:32:11.382 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-15T20:32:11.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T20:32:11.677 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T20:32:11.690 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T20:32:11.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:32:11.810 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-15T20:32:11.810 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:32:11.811 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-15T20:32:11.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T20:32:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T20:32:12.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:12.455 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:12.455 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:12.469 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-15T20:32:11.622726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1516880,"kb_used_data":1276276,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375970480,"statfs":{"total":386547056640,"available":384993771520,"internally_reserved":0,"allocated":1306906624,"data_stored":20887309504,"data_compressed":30571162,"data_compressed_allocated":1305415680,"data_compressed_original":20886650880,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":729808896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001430"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":32774,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:05.912180+0000","last_change":"2024-09-15T20:32:05.912076+0000","last_active":"2024-09-15T20:32:05.912180+0000","last_peered":"2024-09-15T20:32:05.912180+0000","last_clean":"2024-09-15T20:32:05.912180+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:05.912180+0000","last_undegraded":"2024-09-15T20:32:05.912180+0000","last_fullsized":"2024-09-15T20:32:05.912180+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:09:11.904454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":76,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:05.907485+0000","last_change":"2024-09-15T20:30:19.365269+0000","last_active":"2024-09-15T20:32:05.907485+0000","last_peered":"2024-09-15T20:32:05.907485+0000","last_clean":"2024-09-15T20:32:05.907485+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:32:05.907485+0000","last_undegraded":"2024-09-15T20:32:05.907485+0000","last_fullsized":"2024-09-15T20:32:05.907485+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:33.583068+0000","last_clean_scrub_stamp":"2024-09-15T20:25:33.583068+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:14.661175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305382912,"data_stored":20886003712,"data_compressed":30569342,"data_compressed_allocated":1305374720,"data_compressed_original":20885995520,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640345,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709124,"kb_used_data":627948,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662716,"statfs":{"total":96636764160,"available":95910621184,"internally_reserved":0,"allocated":643018752,"data_stored":10282502704,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95584,"kb_used_data":19976,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276256,"statfs":{"total":96636764160,"available":96538886144,"internally_reserved":0,"allocated":20455424,"data_stored":321489456,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684756,"kb_used_data":627968,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93687084,"statfs":{"total":96636764160,"available":95935574016,"internally_reserved":0,"allocated":643039232,"data_stored":10282826800,"data_compressed":14906030,"data_compressed_allocated":642666496,"data_compressed_original":10282663936,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":393216,"data_stored":490544,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":642646016,"data_stored":10282336256,"data_compressed":14905120,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:12.473 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T20:25:46.631534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=25, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:32:12.473 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T20:25:33.583068+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=25, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:32:12.473 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T20:32:27.581 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-09-15T20:32:27.578+0000 7f4072b10640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-15T20:32:32.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:32.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:32.823 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:32.836 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-15T20:32:31.625344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1516868,"kb_used_data":1276276,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375970492,"statfs":{"total":386547056640,"available":384993783808,"internally_reserved":0,"allocated":1306906624,"data_stored":20887309504,"data_compressed":30571162,"data_compressed_allocated":1305415680,"data_compressed_original":20886650880,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":33026,"reported_epoch":44,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-15T20:32:31.189405+0000","last_change":"2024-09-15T20:32:12.244182+0000","last_active":"2024-09-15T20:32:31.189405+0000","last_peered":"2024-09-15T20:32:31.189405+0000","last_clean":"2024-09-15T20:32:31.189405+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:31.189405+0000","last_undegraded":"2024-09-15T20:32:31.189405+0000","last_fullsized":"2024-09-15T20:32:31.189405+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:25:46.631534+0000","last_clean_scrub_stamp":"2024-09-15T20:25:46.631534+0000","objects_scrubbed":3704,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'19","reported_seq":86,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:12.113379+0000","last_change":"2024-09-15T20:32:12.113379+0000","last_active":"2024-09-15T20:32:12.113379+0000","last_peered":"2024-09-15T20:32:12.113379+0000","last_clean":"2024-09-15T20:32:12.113379+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:32:12.113379+0000","last_undegraded":"2024-09-15T20:32:12.113379+0000","last_fullsized":"2024-09-15T20:32:12.113379+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'19","last_scrub_stamp":"2024-09-15T20:32:12.113309+0000","last_deep_scrub":"44'19","last_deep_scrub_stamp":"2024-09-15T20:32:12.113309+0000","last_clean_scrub_stamp":"2024-09-15T20:32:12.113309+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:13.243917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305382912,"data_stored":20886003712,"data_compressed":30569342,"data_compressed_allocated":1305374720,"data_compressed_original":20885995520,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640349,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709124,"kb_used_data":627948,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662716,"statfs":{"total":96636764160,"available":95910621184,"internally_reserved":0,"allocated":643018752,"data_stored":10282502704,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95584,"kb_used_data":19976,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276256,"statfs":{"total":96636764160,"available":96538886144,"internally_reserved":0,"allocated":20455424,"data_stored":321489456,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691881,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684752,"kb_used_data":627968,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93687088,"statfs":{"total":96636764160,"available":95935578112,"internally_reserved":0,"allocated":643039232,"data_stored":10282826800,"data_compressed":14906030,"data_compressed_allocated":642666496,"data_compressed_original":10282663936,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705755,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":490544,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":642646016,"data_stored":10282336256,"data_compressed":14905120,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:32.837 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T20:25:46.631534+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=25, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:32:32.837 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T20:32:52.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:32:53.132 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-15T20:32:53.132 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-15T20:32:53.144 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-15T20:32:51.629431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20565000728,"num_objects":4906,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":22897,"num_write_kb":56852804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1516860,"kb_used_data":1276276,"kb_used_omap":8,"kb_used_meta":240503,"kb_avail":375970500,"statfs":{"total":386547056640,"available":384993792000,"internally_reserved":0,"allocated":1306906624,"data_stored":20887309504,"data_compressed":30571162,"data_compressed_allocated":1305415680,"data_compressed_original":20886650880,"omap_allocated":9187,"internal_metadata":246275101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"2.0","version":"23'22860","reported_seq":33110,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:37.295240+0000","last_change":"2024-09-15T20:32:37.295240+0000","last_active":"2024-09-15T20:32:37.295240+0000","last_peered":"2024-09-15T20:32:37.295240+0000","last_clean":"2024-09-15T20:32:37.295240+0000","last_became_active":"2024-09-15T20:32:05.911603+0000","last_became_peered":"2024-09-15T20:32:05.911603+0000","last_unstale":"2024-09-15T20:32:37.295240+0000","last_undegraded":"2024-09-15T20:32:37.295240+0000","last_fullsized":"2024-09-15T20:32:37.295240+0000","mapping_epoch":43,"log_start":"23'22833","ondisk_log_start":"23'22833","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'22860","last_scrub_stamp":"2024-09-15T20:32:37.295147+0000","last_deep_scrub":"23'22860","last_deep_scrub_stamp":"2024-09-15T20:32:37.295147+0000","last_clean_scrub_stamp":"2024-09-15T20:32:37.295147+0000","objects_scrubbed":4904,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:20:29.855315+0000","scrub_duration":25050,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"44'19","reported_seq":86,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T20:32:12.113379+0000","last_change":"2024-09-15T20:32:12.113379+0000","last_active":"2024-09-15T20:32:12.113379+0000","last_peered":"2024-09-15T20:32:12.113379+0000","last_clean":"2024-09-15T20:32:12.113379+0000","last_became_active":"2024-09-15T20:30:19.364925+0000","last_became_peered":"2024-09-15T20:30:19.364925+0000","last_unstale":"2024-09-15T20:32:12.113379+0000","last_undegraded":"2024-09-15T20:32:12.113379+0000","last_fullsized":"2024-09-15T20:32:12.113379+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'19","last_scrub_stamp":"2024-09-15T20:32:12.113309+0000","last_deep_scrub":"44'19","last_deep_scrub_stamp":"2024-09-15T20:32:12.113309+0000","last_clean_scrub_stamp":"2024-09-15T20:32:12.113309+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:13.243917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":20564672536,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22860,"num_write_kb":56852482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4904,"num_bytes_recovered":20564672536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305382912,"data_stored":20886003712,"data_compressed":30569342,"data_compressed_allocated":1305374720,"data_compressed_original":20885995520,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1820,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709116,"kb_used_data":627948,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93662724,"statfs":{"total":96636764160,"available":95910629376,"internally_reserved":0,"allocated":643018752,"data_stored":10282502704,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95584,"kb_used_data":19976,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94276256,"statfs":{"total":96636764160,"available":96538886144,"internally_reserved":0,"allocated":20455424,"data_stored":321489456,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684752,"kb_used_data":627968,"kb_used_omap":4,"kb_used_meta":56763,"kb_avail":93687088,"statfs":{"total":96636764160,"available":95935578112,"internally_reserved":0,"allocated":643039232,"data_stored":10282826800,"data_compressed":14906030,"data_compressed_allocated":642666496,"data_compressed_original":10282663936,"omap_allocated":4404,"internal_metadata":58126028},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705759,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":490544,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":910,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":642646016,"data_stored":10282336256,"data_compressed":14905120,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20086784,"data_stored":321327104,"data_compressed":614502,"data_compressed_allocated":20082688,"data_compressed_original":321323008,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":642650112,"data_stored":10282340352,"data_compressed":15049720,"data_compressed_allocated":642646016,"data_compressed_original":10282336256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:32:53.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T20:32:53.442 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T20:32:53.443 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T20:32:53.443 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T20:32:53.443 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.472 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T20:32:53.473 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T20:32:53.473 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.542 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T20:32:53.542 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T20:32:53.542 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.610 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T20:32:53.610 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T20:32:53.610 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.694 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T20:32:53.694 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T20:32:53.695 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T20:32:53.695 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.721 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T20:32:53.721 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T20:32:53.721 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T20:32:53.721 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.771 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T20:32:53.772 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T20:32:53.772 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.823 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T20:32:53.823 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T20:32:53.823 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:32:53.874 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T20:32:53.874 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T20:32:53.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v Error | egrep -v 'overall HEALTH_' | egrep -v '\(OBJECT_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-15T20:32:53.946 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:32:53.946 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T20:32:54.190 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:32:54.190 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T20:32:54.298 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:32:54.298 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T20:32:54.366 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:32:54.366 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T20:32:54.426 INFO:tasks.ceph:Archiving mon data... 2024-09-15T20:32:54.427 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604/data/mon.a.tgz 2024-09-15T20:32:54.427 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-15T20:32:54.442 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.cAc9fsAJbi 2024-09-15T20:32:54.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.cAc9fsAJbi 2024-09-15T20:32:54.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.cAc9fsAJbi 2024-09-15T20:32:54.672 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.cAc9fsAJbi is 340KB 2024-09-15T20:32:54.739 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.cAc9fsAJbi 2024-09-15T20:32:54.755 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604/data/mon.b.tgz 2024-09-15T20:32:54.755 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-15T20:32:54.810 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.kE7iuc9NcU 2024-09-15T20:32:54.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.kE7iuc9NcU 2024-09-15T20:32:54.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.kE7iuc9NcU 2024-09-15T20:32:55.027 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.kE7iuc9NcU is 341KB 2024-09-15T20:32:55.121 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.kE7iuc9NcU 2024-09-15T20:32:55.136 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604/data/mon.c.tgz 2024-09-15T20:32:55.136 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-15T20:32:55.189 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.2BvKxdHeqI 2024-09-15T20:32:55.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2BvKxdHeqI 2024-09-15T20:32:55.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.2BvKxdHeqI 2024-09-15T20:32:55.390 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.2BvKxdHeqI is 264KB 2024-09-15T20:32:55.482 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.2BvKxdHeqI 2024-09-15T20:32:55.497 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T20:32:55.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T20:32:55.616 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-15T20:32:55.616 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T20:32:55.618 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604/remote/smithi129/crash 2024-09-15T20:32:55.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T20:32:55.680 INFO:tasks.ceph:Compressing logs... 2024-09-15T20:32:55.681 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T20:32:55.745 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T20:32:55.746 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37621.log 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47090.log 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48226.log 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37621.log.gz 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47090.log.gz 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45662.log 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33502.log 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48226.log.gz 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46544.log 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45662.log.gz 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.33502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-09-15T20:32:55.859 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33502.log.gz 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46544.log.gz 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40717.log 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40718.log 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42105.log 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40717.log.gz 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40718.log.gz 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2024-09-15T20:32:55.860 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42105.log.gz 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.33384.log: gzip 0.0% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz /var/log/ceph/ceph-client.admin.44355.log 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44355.log.gz 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47174.log 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45028.log 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39178.log 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47174.log.gz 2024-09-15T20:32:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38109.log 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45028.log.gz 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.39178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38903.log 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39178.log.gz 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38109.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38109.log.gz -5 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47552.log 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41120.log 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38903.log.gz 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47552.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.47552.log.gz 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.40903.log 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44313.log 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41120.log.gz 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45323.log 2024-09-15T20:32:55.862 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46460.log 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44313.log.gz 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45323.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45323.log.gz 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44481.log 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46460.log.gz 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43221.log 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43935.log 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44481.log.gz 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43221.log.gz 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45830.log 2024-09-15T20:32:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43935.log.gz 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38987.log 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41936.log 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45830.log.gz 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38987.log.gz 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47846.log 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41936.log.gz 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44902.log 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47846.log.gz 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44986.log 2024-09-15T20:32:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42480.log 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44902.log.gz 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44986.log.gz 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48182.log 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37642.log 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42480.log.gz 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47342.log 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48182.log.gz 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37642.log.gz 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27224.log 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47342.log.gz 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.tmp-client.admin.27224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39272.log 2024-09-15T20:32:55.865 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27224.log.gz 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44187.log 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.39272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44859.log 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39272.log.gz 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46040.log 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44187.log.gz 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46796.log 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44859.log.gz 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46040.log.gz 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37980.log 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46796.log.gz 2024-09-15T20:32:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40364.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz -5 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42103.log 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41834.log 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37980.log.gz 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46712.log 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42103.log.gz 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37494.log 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41834.log.gz 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46586.log 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46712.log.gz 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37494.log.gz 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46376.log 2024-09-15T20:32:55.867 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46586.log.gz 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44103.log 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38066.log 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46376.log.gz 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33592.log 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44103.log.gz 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45154.log 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38066.log.gz 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.33592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38252.log 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33592.log.gz 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38380.log 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45154.log.gz 2024-09-15T20:32:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38252.log.gz 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46922.log 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38380.log.gz 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46922.log.gz 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44019.log 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43557.log 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44019.log.gz 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.39113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47930.log 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2024-09-15T20:32:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38568.log 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43557.log.gz 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47132.log 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47930.log.gz 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38568.log.gz 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45494.log 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33344.log 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47132.log.gz 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45239.log 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45494.log.gz 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.33344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41934.log 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33344.log.gz 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46166.log 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45239.log.gz 2024-09-15T20:32:55.870 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43683.log 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41934.log.gz 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40866.log 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46166.log.gz 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43683.log.gz 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40866.log.gz 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44397.log 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43137.log 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47594.log 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44397.log.gz 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43137.log.gz 2024-09-15T20:32:55.871 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40549.log 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44229.log 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47594.log.gz 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40549.log.gz 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44565.log 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44229.log.gz 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48268.log 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44565.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.45704.log 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44565.log.gz 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39155.log 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48268.log.gz 2024-09-15T20:32:55.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45704.log.gz 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.39155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38823.log 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39155.log.gz 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48014.log 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47216.log 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38823.log.gz 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48014.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48014.log.gz 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40994.log 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47216.log.gz 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45914.log 2024-09-15T20:32:55.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40994.log.gz 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44061.log 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45914.log.gz 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39071.log 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44061.log.gz 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.39215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38022.log 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46754.log 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.39071.log: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39071.log.gz 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45872.log 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38022.log.gz 2024-09-15T20:32:55.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38569.log 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46754.log.gz 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45872.log.gz 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45452.log 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45366.log 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38569.log.gz 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44145.log 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45452.log.gz 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45366.log.gz 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41220.log 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44145.log.gz 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2024-09-15T20:32:55.875 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45410.log 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41220.log.gz 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42481.log 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43641.log 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45410.log.gz 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47804.log 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42481.log.gz 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43641.log.gz 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45998.log 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47804.log.gz 2024-09-15T20:32:55.876 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45956.log 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37938.log 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45998.log.gz 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45956.log.gz 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46838.log 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43305.log 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37938.log.gz 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47888.log 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46838.log.gz 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43179.log 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43305.log.gz 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42927.log 2024-09-15T20:32:55.877 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47888.log.gz 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42650.log 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43179.log.gz 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42927.log.gz 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37812.log 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42650.log.gz 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45112.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38151.log 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37812.log.gz 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42985.log 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-09-15T20:32:55.878 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38151.log.gz 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42985.log.gz 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33304.log 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46334.log 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.33304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33304.log.gz 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47972.log 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40448.log 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46334.log.gz 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47972.log.gz 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40448.log.gz 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47006.log 2024-09-15T20:32:55.879 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47006.log.gz 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37040.log 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41387.log 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37040.log.gz 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41387.log.gz 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44775.log 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40547.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz -5 2024-09-15T20:32:55.880 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45788.log 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43851.log 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44775.log.gz 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45788.log.gz 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42104.log 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42278.log 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43851.log.gz 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42104.log.gz 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37098.log 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42278.log.gz 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2024-09-15T20:32:55.881 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41937.log 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41666.log 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37098.log.gz 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41937.log.gz 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37702.log 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38253.log 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41666.log.gz 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37702.log.gz 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38381.log 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38253.log.gz 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38381.log.gz 2024-09-15T20:32:55.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48308.log 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39314.log 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48308.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.48308.log.gz 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.41560.log 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.39314.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39314.log.gz 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41560.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33264.log 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41560.log.gz 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42986.log 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.33264.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.33264.log.gz --verbose 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.47426.log 2024-09-15T20:32:55.883 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47384.log 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42986.log.gz 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47426.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47426.log.gz 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43347.log 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47384.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47384.log.gz -5 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38945.log 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43347.log.gz 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38697.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 0.0% 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46124.log 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38945.log.gz 2024-09-15T20:32:55.884 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37224.log 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36933.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46124.log.gz 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37224.log.gz 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46880.log 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36933.log.gz 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37182.log 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46880.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.42649.log 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46880.log.gz 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38529.log 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37182.log.gz 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5/var/log/ceph/ceph-client.admin.42649.log: --verbose -- /var/log/ceph/ceph-client.admin.42233.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42649.log.gz 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:32:55.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38529.log.gz 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42987.log 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37324.log 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42987.log.gz 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37854.log 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37324.log.gz 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42818.log 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37854.log.gz 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37896.log 2024-09-15T20:32:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42818.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48140.log 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42818.log.gz 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47048.log 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37896.log.gz 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47300.log 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48140.log: /var/log/ceph/ceph-client.admin.47048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46208.log 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47048.log.gz 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48140.log.gz 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47300.log.gz 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43767.log 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46208.log.gz 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45536.log 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43767.log.gz 2024-09-15T20:32:55.887 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45536.log.gz 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44271.log 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39356.log 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44271.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44271.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.37770.log 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42759.log 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37770.log: /var/log/ceph/ceph-client.admin.39356.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39356.log.gz 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37770.log.gz 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48098.log 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42759.log.gz 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37323.log 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr: 94.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T20:32:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41219.log 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48098.log: /var/log/ceph/ceph-client.admin.37323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48098.log.gz 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37323.log.gz 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41219.log.gz 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37326.log 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40677.log: /var/log/ceph/ceph-client.admin.37326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44439.log 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37326.log.gz 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37491.log 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44649.log 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44439.log.gz 2024-09-15T20:32:55.889 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37491.log.gz 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46418.log 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44733.log 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46418.log: /var/log/ceph/ceph-client.admin.44649.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44649.log.gz 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46418.log.gz 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47762.log 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47510.log 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44733.log.gz 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47762.log.gz 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42276.log 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46082.log 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47510.log: /var/log/ceph/ceph-client.admin.42276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42276.log.gz 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47510.log.gz 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44607.log 2024-09-15T20:32:55.890 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46082.log.gz 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47258.log 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43431.log 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47258.log: /var/log/ceph/ceph-client.admin.44607.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47258.log.gz 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44607.log.gz 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43389.log 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47678.log 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45746.log 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47678.log: /var/log/ceph/ceph-client.admin.43389.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47678.log.gz 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43389.log.gz 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42213.log 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45746.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-09-15T20:32:55.891 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42817.log 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42213.log.gz 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.39029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42817.log.gz 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46670.log 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42819.log 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.46670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46670.log.gz 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37493.log 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42819.log.gz 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43725.log 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37493.log.gz 2024-09-15T20:32:55.892 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43809.log 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43725.log.gz 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45620.log 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43809.log.gz 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46250.log 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45620.log.gz 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46964.log 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46250.log.gz 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45196.log 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46964.log.gz 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44691.log 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45196.log.gz 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43263.log 2024-09-15T20:32:55.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44691.log.gz 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32951.log 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43263.log.gz 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43599.log 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.32951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32951.log.gz 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43515.log 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43599.log.gz 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45281.log 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43515.log.gz 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47636.log 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45281.log.gz 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41498.log 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47636.log.gz 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45578.log 2024-09-15T20:32:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33127.log 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45578.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40865.log 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.33127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33127.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44523.log 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40865.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42381.log 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44523.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38250.log 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42381.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38846.log 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38250.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38846.log.gz 2024-09-15T20:32:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40406.log 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44817.log 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40406.log.gz 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38251.log 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44817.log.gz 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47468.log 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38251.log.gz 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43473.log 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47468.log.gz 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43893.log 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43473.log.gz 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2024-09-15T20:32:55.896 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43893.log.gz 2024-09-15T20:32:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T20:32:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2024-09-15T20:32:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37325.log 2024-09-15T20:32:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38421.log 2024-09-15T20:32:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.37325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37325.log.gz 2024-09-15T20:32:55.906 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47720.log 2024-09-15T20:32:55.906 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38421.log.gz 2024-09-15T20:32:55.906 INFO:teuthology.orchestra.run.smithi129.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T20:32:55.906 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38781.log 2024-09-15T20:32:55.906 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47720.log.gz 2024-09-15T20:32:55.906 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40826.log 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38781.log.gz 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48056.log 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.40826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40826.log.gz 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45070.log 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48056.log.gz 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42651.log 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45070.log: /var/log/ceph/ceph-client.admin.43977.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45070.log.gz 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46502.log 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46628.log 2024-09-15T20:32:55.907 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42651.log.gz 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46502.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41728.log 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38420.log 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41728.log: /var/log/ceph/ceph-client.admin.46628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46628.log.gz 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41728.log.gz 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42483.log 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.38420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38420.log.gz 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42483.log.gz 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46292.log 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz/var/log/ceph/ceph-client.admin.46292.log: 2024-09-15T20:32:55.908 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46292.log.gz 2024-09-15T20:32:55.933 INFO:teuthology.orchestra.run.smithi129.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T20:32:56.001 INFO:teuthology.orchestra.run.smithi129.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T20:32:56.063 INFO:teuthology.orchestra.run.smithi129.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T20:33:02.718 INFO:teuthology.orchestra.run.smithi129.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T20:33:20.253 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T20:33:42.774 INFO:teuthology.orchestra.run.smithi129.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T20:33:42.775 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-15T20:33:42.775 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m47.039s 2024-09-15T20:33:42.775 INFO:teuthology.orchestra.run.smithi129.stderr:user 1m16.430s 2024-09-15T20:33:42.775 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m2.789s 2024-09-15T20:33:42.776 INFO:tasks.ceph:Archiving logs... 2024-09-15T20:33:42.776 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604/remote/smithi129/log 2024-09-15T20:33:42.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T20:33:55.204 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T20:33:55.213 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T20:33:55.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T20:33:55.273 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T20:33:55.282 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T20:33:55.282 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:33:55.300 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-15T20:33:55.305 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T20:33:55.305 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-15T20:33:55.305 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +449us[ +432us] +/- 44ms 2024-09-15T20:33:55.305 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 +201us[ +201us] +/- 71ms 2024-09-15T20:33:55.305 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -927us[ -944us] +/- 65ms 2024-09-15T20:33:55.305 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:33:55.306 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T20:33:55.314 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T20:33:55.315 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T20:33:55.333 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T20:33:55.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T20:33:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:33:55.445 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T20:33:55.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T20:33:55.476 INFO:teuthology.task.internal:Duration was 967.346785 seconds 2024-09-15T20:33:55.477 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T20:33:55.501 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T20:33:55.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T20:33:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T20:33:55.872 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T20:33:55.872 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-09-15T20:33:55.872 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T20:33:55.933 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T20:33:55.933 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T20:33:55.999 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T20:33:55.999 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T20:33:56.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T20:33:56.158 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T20:33:56.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T20:33:56.183 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T20:33:56.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:33:56.247 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-09-15T20:33:56.277 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:33:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:33:56.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T20:33:56.341 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T20:33:56.342 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905604/remote/smithi129 2024-09-15T20:33:56.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T20:33:56.406 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T20:33:56.406 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T20:33:56.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T20:33:56.466 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T20:33:56.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T20:33:56.486 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T20:33:56.486 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T20:33:56.510 INFO:teuthology.orchestra.run.smithi129.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 20:33 /home/ubuntu/cephtest 2024-09-15T20:33:56.512 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T20:33:56.522 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T20:33:56.578 INFO:teuthology.run:Summary data: description: rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest}} duration: 967.3467845916748 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-15T20:33:56.579 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T20:33:56.664 INFO:teuthology.run:pass