2024-09-11T04:09:21.192 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T04:09:21.207 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T04:09:21.292 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900128 branch: wip-hemanth-testing-2024-09-10-0723 description: rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} email: hyelloji@redhat.com first_in_suite: false job_id: '7900128' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi no_nested_subset: false os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 seed: 9167 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOpSGdH+8trOW0p0FG1PL5/udYbLGkylBP7EU+5bi0y8OzHGBvQknyFiVwRmn+Rphynhnt2pVRmEKHDzLhNgIIw= tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: all: - rados/test_health_warnings.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:16:14 tube: smithi user: hyelloji verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T04:09:21.292 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-11T04:09:21.293 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-11T04:09:21.293 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T04:09:21.295 INFO:teuthology.task.internal:Checking packages... 2024-09-11T04:09:21.316 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8f219c476cb4f26fc5719c661c0323719a45cb30' 2024-09-11T04:09:21.316 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T04:09:21.317 INFO:teuthology.packaging:ref: None 2024-09-11T04:09:21.317 INFO:teuthology.packaging:tag: None 2024-09-11T04:09:21.317 INFO:teuthology.packaging:branch: wip-hemanth-testing-2024-09-10-0723 2024-09-11T04:09:21.317 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:09:21.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-hemanth-testing-2024-09-10-0723 2024-09-11T04:09:21.463 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4854-g8f219c47-1jammy 2024-09-11T04:09:21.465 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T04:09:21.471 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T04:09:21.471 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T04:09:21.513 INFO:teuthology.task.internal:Saving configuration 2024-09-11T04:09:21.523 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T04:09:21.581 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T04:09:21.606 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900128', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 04:04:00.489266', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFhGA78UT66vvhZzFJu1aF1PGu/Jjqm0JzcFXzi4q4Rs'} 2024-09-11T04:09:21.606 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T04:09:21.612 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2024-09-11T04:09:21.612 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T04:09:21.670 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f37dd283be0>, signals=[15]) 2024-09-11T04:09:21.671 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T04:09:21.676 INFO:teuthology.task.internal:Opening connections... 2024-09-11T04:09:21.677 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:09:21.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:09:21.758 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T04:09:21.764 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-09-11T04:09:21.878 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-09-11T04:09:21.879 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-09-11T04:09:21.926 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T04:09:21.926 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Ubuntu" 2024-09-11T04:09:21.926 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="22.04" 2024-09-11T04:09:21.926 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T04:09:21.926 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_CODENAME=jammy 2024-09-11T04:09:21.926 INFO:teuthology.orchestra.run.smithi002.stdout:ID=ubuntu 2024-09-11T04:09:21.926 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE=debian 2024-09-11T04:09:21.927 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T04:09:21.927 INFO:teuthology.orchestra.run.smithi002.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T04:09:21.927 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T04:09:21.927 INFO:teuthology.orchestra.run.smithi002.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T04:09:21.927 INFO:teuthology.orchestra.run.smithi002.stdout:UBUNTU_CODENAME=jammy 2024-09-11T04:09:21.927 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-09-11T04:09:21.947 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T04:09:21.955 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T04:09:21.973 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T04:09:21.973 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-09-11T04:09:21.978 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T04:09:21.997 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T04:09:21.997 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-09-11T04:09:22.027 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T04:09:22.055 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T04:09:22.081 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T04:09:22.081 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T04:09:22.081 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T04:09:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:09:22.086 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-09-11T04:09:22.134 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-56-generic 2024-09-11T04:09:22.135 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.15.0-56-generic 2024-09-11T04:09:22.135 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2024-09-11T04:09:22.260 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2024-09-11T04:09:22.400 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T04:09:22.409 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T04:09:22.474 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T04:09:22.509 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T04:09:22.668 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T04:09:22.736 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T04:09:22.772 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T04:09:22.776 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T04:09:22.776 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T04:09:22.796 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T04:09:22.796 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T04:09:22.800 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T04:09:22.801 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T04:09:22.807 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T04:09:22.815 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T04:09:22.817 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T04:09:22.843 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T04:09:22.844 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T04:09:22.844 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T04:09:22.845 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T04:09:22.845 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T04:09:22.845 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T04:09:22.846 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T04:09:22.846 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T04:09:22.861 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T04:09:22.878 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T04:09:22.879 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T04:09:22.879 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T04:09:22.956 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T04:09:23.373 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T04:09:23.389 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T04:09:23.398 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T04:09:23.398 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T04:09:23.400 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T04:09:23.496 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T04:09:23.507 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T04:09:23.507 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T04:09:23.530 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T04:09:23.549 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T04:09:23.555 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T04:09:23.556 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T04:09:23.556 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T04:09:23.557 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T04:09:23.635 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T04:09:29.543 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 15.8 MB in 2s (7,152 kB/s) 2024-09-11T04:09:31.071 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-09-11T04:09:31.089 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T04:09:31.157 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-09-11T04:09:31.353 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2024-09-11T04:09:31.354 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2024-09-11T04:09:31.526 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T04:09:31.526 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T04:09:31.526 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T04:09:31.526 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T04:09:31.526 INFO:teuthology.orchestra.run.smithi002.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T04:09:31.527 INFO:teuthology.orchestra.run.smithi002.stdout: python2.7-minimal 2024-09-11T04:09:31.527 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T04:09:31.528 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2024-09-11T04:09:31.528 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T04:09:31.528 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T04:09:31.528 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T04:09:31.529 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2024-09-11T04:09:31.529 INFO:teuthology.orchestra.run.smithi002.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T04:09:31.530 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2024-09-11T04:09:31.530 INFO:teuthology.orchestra.run.smithi002.stdout: thermald 2024-09-11T04:09:31.580 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2024-09-11T04:09:31.580 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T04:09:31.581 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T04:09:31.581 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T04:09:31.582 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages will be upgraded: 2024-09-11T04:09:31.582 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T04:09:31.641 INFO:teuthology.orchestra.run.smithi002.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T04:09:31.641 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 113 MB of archives. 2024-09-11T04:09:31.641 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T04:09:31.641 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T04:09:32.073 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T04:09:32.170 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T04:09:33.150 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T04:09:33.150 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T04:09:33.150 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T04:09:33.283 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T04:09:33.306 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T04:09:33.737 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 113 MB in 2s (66.0 MB/s) 2024-09-11T04:09:33.883 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T04:09:35.832 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T04:09:35.836 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:09:35.989 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:09:38.173 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T04:09:38.195 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:09:38.286 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:09:38.751 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T04:09:38.772 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:09:38.809 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:09:46.028 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:09:46.154 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:09:46.505 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:09:46.607 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:09:46.919 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T04:09:46.941 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T04:09:47.010 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T04:09:53.516 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T04:09:53.553 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:09:53.599 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:09:56.429 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:09:56.537 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:09:56.862 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T04:09:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:09:57.134 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T04:09:57.263 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:09:58.706 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:09:58.706 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T04:09:58.706 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:09:58.707 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T04:09:58.915 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:10:00.416 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:10:01.860 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T04:10:01.985 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T04:10:02.120 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:10:02.177 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T04:10:02.177 INFO:teuthology.orchestra.run.smithi002.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:10:16.666 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T04:10:16.667 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub' 2024-09-11T04:10:16.697 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T04:10:16.721 INFO:teuthology.orchestra.run.smithi002.stdout:Generating grub configuration file ... 2024-09-11T04:10:17.029 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:10:17.047 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:10:17.379 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:10:17.385 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T04:10:17.458 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T04:10:17.464 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T04:10:17.625 INFO:teuthology.orchestra.run.smithi002.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T04:10:17.625 INFO:teuthology.orchestra.run.smithi002.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T04:10:17.625 INFO:teuthology.orchestra.run.smithi002.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T04:10:17.659 INFO:teuthology.orchestra.run.smithi002.stdout:done 2024-09-11T04:10:18.163 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2024-09-11T04:10:18.189 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 21 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.119.119 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T04:10:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2024-09-11T04:10:18.191 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2024-09-11T04:10:18.191 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2024-09-11T04:10:18.191 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2024-09-11T04:10:18.191 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T04:10:18.191 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T04:10:18.191 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T04:10:18.191 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T04:10:18.191 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T04:10:18.192 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2024-09-11T04:10:18.349 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2024-09-11T04:10:18.479 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T04:10:18.590 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T04:10:18.675 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T04:10:18.753 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T04:10:19.963 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-09-11T04:10:19.982 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T04:10:20.047 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-09-11T04:10:20.258 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2024-09-11T04:10:20.259 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2024-09-11T04:10:20.454 INFO:teuthology.orchestra.run.smithi002.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T04:10:20.455 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T04:10:20.455 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T04:10:20.455 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T04:10:20.455 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T04:10:20.455 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T04:10:20.456 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T04:10:20.456 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T04:10:20.498 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T04:10:20.500 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2024-09-11T04:10:20.516 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2024-09-11T04:10:20.516 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2024-09-11T04:10:20.516 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2024-09-11T04:10:20.516 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2024-09-11T04:10:20.516 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 21 2024-09-11T04:10:20.516 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T04:10:20.516 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.119.119 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2024-09-11T04:10:20.517 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2024-09-11T04:10:20.518 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-09-11T04:10:20.565 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.P0Yva3uXVp 2024-09-11T04:10:20.565 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub/grub.cfg /tmp/tmp.P0Yva3uXVp 2024-09-11T04:10:20.620 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.P0Yva3uXVp 2024-09-11T04:10:20.741 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.P0Yva3uXVp is 10KB 2024-09-11T04:10:20.754 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.P0Yva3uXVp 2024-09-11T04:10:20.760 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T04:10:20.813 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:10:20.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T04:10:20.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T04:10:20.892 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T04:10:20.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo update-grub 2024-09-11T04:10:22.274 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub' 2024-09-11T04:10:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T04:10:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-09-11T04:10:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:10:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:10:22.760 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:10:22.765 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T04:10:22.810 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T04:10:22.815 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T04:10:22.953 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T04:10:22.953 INFO:teuthology.orchestra.run.smithi002.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T04:10:22.953 INFO:teuthology.orchestra.run.smithi002.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T04:10:22.973 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-09-11T04:10:22.982 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-09-11T04:10:52.996 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T04:10:52.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-11T04:10:52.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:11:11.552 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2024-09-11T04:11:20.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-11T04:11:20.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:12:20.622 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T04:12:32.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-11T04:12:32.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:12:32.826 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-09-11T04:12:33.514 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-11T04:12:33.514 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-11T04:12:33.514 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-09-11T04:12:33.563 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-119-generic 2024-09-11T04:12:33.563 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T04:12:33.563 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T04:12:33.563 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T04:12:34.565 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T04:12:34.565 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T04:12:34.658 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-09-11T04:12:34.676 DEBUG:teuthology.parallel:result is None 2024-09-11T04:12:34.676 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T04:12:34.684 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T04:12:34.684 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T04:12:34.690 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T04:12:34.711 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T04:12:34.745 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T04:12:34.745 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T04:12:34.759 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T04:12:34.780 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T04:12:34.780 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T04:12:34.817 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:12:34.827 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:12:34.829 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T04:12:34.836 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T04:12:34.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T04:12:34.882 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T04:12:34.891 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T04:12:34.891 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T04:12:34.930 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T04:12:34.978 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T04:12:35.027 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:12:35.027 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T04:12:35.096 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-09-11T04:12:35.177 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T04:12:35.189 INFO:teuthology.task.internal:Starting timer... 2024-09-11T04:12:35.189 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T04:12:35.257 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T04:12:35.289 DEBUG:teuthology.task.selinux:Excluding smithi002: OS 'ubuntu' does not support SELinux 2024-09-11T04:12:35.289 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T04:12:35.290 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-11T04:12:35.290 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T04:12:35.290 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T04:12:35.322 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T04:12:35.325 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T04:12:35.433 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T04:12:35.444 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T04:12:35.446 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T04:20:09.826 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com')] 2024-09-11T04:20:09.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-11T04:20:09.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:20:09.912 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-09-11T04:20:10.001 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-11T04:20:10.001 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T04:20:10.015 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T04:20:10.015 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T04:20:10.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T04:20:10.073 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T04:20:10.073 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Command line: ntpd -gq 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: ---------------------------------------------------- 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: corporation. Support and training for ntp-4 are 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: available at https://www.nwtime.org/support 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: ---------------------------------------------------- 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: proto: precision = 0.046 usec (-24) 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: basedate set to 2022-02-04 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: gps base set to 2022-02-06 (week 2196) 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T04:20:10.074 INFO:teuthology.orchestra.run.smithi002.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T04:20:10.075 INFO:teuthology.orchestra.run.smithi002.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T04:20:10.075 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T04:20:10.076 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T04:20:10.076 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T04:20:10.076 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T04:20:10.076 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Listen normally on 3 ens1f0 172.21.15.2:123 2024-09-11T04:20:10.076 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Listen normally on 4 lo [::1]:123 2024-09-11T04:20:10.076 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1920%4]:123 2024-09-11T04:20:10.076 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:10 ntpd[18362]: Listening on routing socket on fd #22 for interface updates 2024-09-11T04:20:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:11 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:12.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:12 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:13 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:14.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:14 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:14.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:14 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:14.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:14 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:14.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:14 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:14.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:14 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:14.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:14 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:15.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:15.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:15.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:15.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:15.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:15.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:15.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:15 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:16.073 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:16 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:16 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:16 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:16 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:16 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:16 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T04:20:17.074 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T04:20:17.075 INFO:teuthology.orchestra.run.smithi002.stdout:11 Sep 04:20:17 ntpd[18362]: ntpd: time slew -0.001074 s 2024-09-11T04:20:17.075 INFO:teuthology.orchestra.run.smithi002.stdout:ntpd: time slew -0.001074s 2024-09-11T04:20:17.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T04:20:17.075 INFO:teuthology.orchestra.run.smithi002.stderr:11 Sep 04:20:17 ntpd[18362]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T04:20:17.129 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T04:20:17.129 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2024-09-11T04:20:17.129 INFO:teuthology.orchestra.run.smithi002.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T04:20:17.129 INFO:teuthology.orchestra.run.smithi002.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T04:20:17.129 INFO:teuthology.orchestra.run.smithi002.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T04:20:17.129 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T04:20:17.130 INFO:teuthology.run_tasks:Running task install... 2024-09-11T04:20:17.139 DEBUG:teuthology.task.install:project ceph 2024-09-11T04:20:17.139 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T04:20:17.139 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T04:20:17.139 INFO:teuthology.task.install:Using flavor: default 2024-09-11T04:20:17.145 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T04:20:17.145 INFO:teuthology.task.install:extra packages: [] 2024-09-11T04:20:17.145 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-key list | grep Ceph 2024-09-11T04:20:17.228 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T04:20:17.255 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T04:20:17.255 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T04:20:17.256 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-11T04:20:17.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:20:17.412 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default/ 2024-09-11T04:20:17.412 INFO:teuthology.task.install.deb:Package version is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T04:20:17.528 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:20:17.528 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T04:20:17.542 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2024-09-11T04:20:17.720 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T04:20:17.729 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T04:20:17.731 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T04:20:17.746 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T04:20:17.773 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T04:20:17.794 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-11T04:20:17.816 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T04:20:17.837 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-11T04:20:17.885 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-09-11T04:20:18.086 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 29.3 kB in 0s (82.5 kB/s) 2024-09-11T04:20:19.111 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-09-11T04:20:19.127 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4854-g8f219c47-1jammy cephadm=19.3.0-4854-g8f219c47-1jammy ceph-mds=19.3.0-4854-g8f219c47-1jammy ceph-mgr=19.3.0-4854-g8f219c47-1jammy ceph-common=19.3.0-4854-g8f219c47-1jammy ceph-fuse=19.3.0-4854-g8f219c47-1jammy ceph-test=19.3.0-4854-g8f219c47-1jammy ceph-volume=19.3.0-4854-g8f219c47-1jammy radosgw=19.3.0-4854-g8f219c47-1jammy python3-rados=19.3.0-4854-g8f219c47-1jammy python3-rgw=19.3.0-4854-g8f219c47-1jammy python3-cephfs=19.3.0-4854-g8f219c47-1jammy python3-rbd=19.3.0-4854-g8f219c47-1jammy libcephfs2=19.3.0-4854-g8f219c47-1jammy libcephfs-dev=19.3.0-4854-g8f219c47-1jammy librados2=19.3.0-4854-g8f219c47-1jammy librbd1=19.3.0-4854-g8f219c47-1jammy rbd-fuse=19.3.0-4854-g8f219c47-1jammy 2024-09-11T04:20:19.191 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-09-11T04:20:19.404 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2024-09-11T04:20:19.404 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2024-09-11T04:20:19.623 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T04:20:19.623 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T04:20:19.623 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T04:20:19.624 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T04:20:19.624 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T04:20:19.624 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T04:20:19.624 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T04:20:19.625 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2024-09-11T04:20:19.625 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T04:20:19.625 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T04:20:19.626 INFO:teuthology.orchestra.run.smithi002.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-11T04:20:19.626 INFO:teuthology.orchestra.run.smithi002.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-11T04:20:19.626 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-11T04:20:19.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet zip 2024-09-11T04:20:19.628 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2024-09-11T04:20:19.629 INFO:teuthology.orchestra.run.smithi002.stdout: btrfs-tools 2024-09-11T04:20:19.737 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2024-09-11T04:20:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T04:20:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T04:20:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T04:20:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T04:20:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-11T04:20:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T04:20:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T04:20:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T04:20:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-11T04:20:19.881 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T04:20:19.881 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 251 MB of archives. 2024-09-11T04:20:19.881 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-11T04:20:19.881 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4854-g8f219c47-1jammy [3,633 kB] 2024-09-11T04:20:19.926 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T04:20:20.075 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4854-g8f219c47-1jammy [3,330 kB] 2024-09-11T04:20:20.142 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4854-g8f219c47-1jammy [721 kB] 2024-09-11T04:20:20.154 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4854-g8f219c47-1jammy [356 kB] 2024-09-11T04:20:20.160 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4854-g8f219c47-1jammy [32.8 kB] 2024-09-11T04:20:20.161 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4854-g8f219c47-1jammy [180 kB] 2024-09-11T04:20:20.165 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4854-g8f219c47-1jammy [71.2 kB] 2024-09-11T04:20:20.166 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4854-g8f219c47-1jammy [338 kB] 2024-09-11T04:20:20.171 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4854-g8f219c47-1jammy [7,988 kB] 2024-09-11T04:20:20.295 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4854-g8f219c47-1jammy [112 kB] 2024-09-11T04:20:20.296 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4854-g8f219c47-1jammy [473 kB] 2024-09-11T04:20:20.303 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4854-g8f219c47-1jammy [28.3 MB] 2024-09-11T04:20:20.346 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T04:20:20.359 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T04:20:20.440 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T04:20:20.669 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T04:20:20.680 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T04:20:20.709 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T04:20:20.718 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-11T04:20:20.720 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T04:20:20.721 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T04:20:20.723 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T04:20:20.740 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T04:20:20.745 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T04:20:20.751 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T04:20:20.832 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T04:20:20.838 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T04:20:20.901 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T04:20:20.902 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4854-g8f219c47-1jammy [5,206 kB] 2024-09-11T04:20:20.906 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T04:20:20.909 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T04:20:20.910 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T04:20:20.911 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T04:20:20.913 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T04:20:20.914 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T04:20:20.915 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T04:20:20.970 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4854-g8f219c47-1jammy [251 kB] 2024-09-11T04:20:20.973 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4854-g8f219c47-1jammy [124 kB] 2024-09-11T04:20:20.974 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4854-g8f219c47-1jammy [1,562 kB] 2024-09-11T04:20:20.988 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4854-g8f219c47-1jammy [6,308 kB] 2024-09-11T04:20:20.996 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T04:20:20.997 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T04:20:20.997 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T04:20:20.998 INFO:teuthology.orchestra.run.smithi002.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T04:20:21.003 INFO:teuthology.orchestra.run.smithi002.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T04:20:21.004 INFO:teuthology.orchestra.run.smithi002.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T04:20:21.077 INFO:teuthology.orchestra.run.smithi002.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4854-g8f219c47-1jammy [23.1 MB] 2024-09-11T04:20:21.078 INFO:teuthology.orchestra.run.smithi002.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T04:20:21.078 INFO:teuthology.orchestra.run.smithi002.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T04:20:21.080 INFO:teuthology.orchestra.run.smithi002.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T04:20:21.081 INFO:teuthology.orchestra.run.smithi002.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T04:20:21.160 INFO:teuthology.orchestra.run.smithi002.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T04:20:21.162 INFO:teuthology.orchestra.run.smithi002.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T04:20:21.164 INFO:teuthology.orchestra.run.smithi002.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T04:20:21.165 INFO:teuthology.orchestra.run.smithi002.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T04:20:21.175 INFO:teuthology.orchestra.run.smithi002.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T04:20:21.176 INFO:teuthology.orchestra.run.smithi002.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T04:20:21.241 INFO:teuthology.orchestra.run.smithi002.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T04:20:21.242 INFO:teuthology.orchestra.run.smithi002.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T04:20:21.245 INFO:teuthology.orchestra.run.smithi002.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T04:20:21.249 INFO:teuthology.orchestra.run.smithi002.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T04:20:21.323 INFO:teuthology.orchestra.run.smithi002.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T04:20:21.325 INFO:teuthology.orchestra.run.smithi002.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T04:20:21.327 INFO:teuthology.orchestra.run.smithi002.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T04:20:21.332 INFO:teuthology.orchestra.run.smithi002.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T04:20:21.333 INFO:teuthology.orchestra.run.smithi002.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T04:20:21.334 INFO:teuthology.orchestra.run.smithi002.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T04:20:21.404 INFO:teuthology.orchestra.run.smithi002.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T04:20:21.531 INFO:teuthology.orchestra.run.smithi002.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T04:20:21.598 INFO:teuthology.orchestra.run.smithi002.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T04:20:21.599 INFO:teuthology.orchestra.run.smithi002.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4854-g8f219c47-1jammy [14.0 kB] 2024-09-11T04:20:21.599 INFO:teuthology.orchestra.run.smithi002.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4854-g8f219c47-1jammy [880 kB] 2024-09-11T04:20:21.601 INFO:teuthology.orchestra.run.smithi002.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T04:20:21.602 INFO:teuthology.orchestra.run.smithi002.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T04:20:21.604 INFO:teuthology.orchestra.run.smithi002.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T04:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4854-g8f219c47-1jammy [2,454 kB] 2024-09-11T04:20:21.648 INFO:teuthology.orchestra.run.smithi002.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4854-g8f219c47-1jammy [813 kB] 2024-09-11T04:20:21.648 INFO:teuthology.orchestra.run.smithi002.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T04:20:21.652 INFO:teuthology.orchestra.run.smithi002.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T04:20:21.652 INFO:teuthology.orchestra.run.smithi002.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T04:20:21.659 INFO:teuthology.orchestra.run.smithi002.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4854-g8f219c47-1jammy [160 kB] 2024-09-11T04:20:21.661 INFO:teuthology.orchestra.run.smithi002.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4854-g8f219c47-1jammy [61.4 MB] 2024-09-11T04:20:21.724 INFO:teuthology.orchestra.run.smithi002.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T04:20:21.724 INFO:teuthology.orchestra.run.smithi002.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T04:20:21.730 INFO:teuthology.orchestra.run.smithi002.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T04:20:21.731 INFO:teuthology.orchestra.run.smithi002.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T04:20:21.731 INFO:teuthology.orchestra.run.smithi002.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T04:20:21.732 INFO:teuthology.orchestra.run.smithi002.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T04:20:21.812 INFO:teuthology.orchestra.run.smithi002.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T04:20:21.813 INFO:teuthology.orchestra.run.smithi002.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T04:20:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T04:20:21.827 INFO:teuthology.orchestra.run.smithi002.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T04:20:21.830 INFO:teuthology.orchestra.run.smithi002.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T04:20:21.832 INFO:teuthology.orchestra.run.smithi002.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T04:20:21.894 INFO:teuthology.orchestra.run.smithi002.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T04:20:21.900 INFO:teuthology.orchestra.run.smithi002.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T04:20:21.903 INFO:teuthology.orchestra.run.smithi002.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T04:20:21.904 INFO:teuthology.orchestra.run.smithi002.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T04:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T04:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T04:20:21.985 INFO:teuthology.orchestra.run.smithi002.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T04:20:21.985 INFO:teuthology.orchestra.run.smithi002.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T04:20:21.987 INFO:teuthology.orchestra.run.smithi002.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T04:20:21.988 INFO:teuthology.orchestra.run.smithi002.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T04:20:22.057 INFO:teuthology.orchestra.run.smithi002.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T04:20:22.059 INFO:teuthology.orchestra.run.smithi002.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T04:20:22.077 INFO:teuthology.orchestra.run.smithi002.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T04:20:22.078 INFO:teuthology.orchestra.run.smithi002.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T04:20:22.139 INFO:teuthology.orchestra.run.smithi002.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T04:20:22.144 INFO:teuthology.orchestra.run.smithi002.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T04:20:22.146 INFO:teuthology.orchestra.run.smithi002.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T04:20:22.938 INFO:teuthology.orchestra.run.smithi002.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4854-g8f219c47-1jammy [8,625 kB] 2024-09-11T04:20:23.050 INFO:teuthology.orchestra.run.smithi002.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4854-g8f219c47-1jammy [14.2 kB] 2024-09-11T04:20:23.050 INFO:teuthology.orchestra.run.smithi002.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4854-g8f219c47-1jammy [55.8 MB] 2024-09-11T04:20:24.218 INFO:teuthology.orchestra.run.smithi002.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4854-g8f219c47-1jammy [130 kB] 2024-09-11T04:20:24.221 INFO:teuthology.orchestra.run.smithi002.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4854-g8f219c47-1jammy [39.0 kB] 2024-09-11T04:20:24.222 INFO:teuthology.orchestra.run.smithi002.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4854-g8f219c47-1jammy [15.9 MB] 2024-09-11T04:20:24.535 INFO:teuthology.orchestra.run.smithi002.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4854-g8f219c47-1jammy [91.2 kB] 2024-09-11T04:20:25.135 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 251 MB in 5s (52.4 MB/s) 2024-09-11T04:20:25.266 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T04:20:25.366 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T04:20:25.371 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T04:20:25.440 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T04:20:25.793 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T04:20:25.820 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T04:20:25.860 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T04:20:26.171 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T04:20:26.198 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T04:20:26.229 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T04:20:26.666 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T04:20:26.693 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T04:20:26.741 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T04:20:27.153 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T04:20:27.180 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T04:20:27.219 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T04:20:27.539 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T04:20:27.566 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T04:20:27.597 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T04:20:27.925 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T04:20:27.952 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T04:20:27.992 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T04:20:28.328 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librados2. 2024-09-11T04:20:28.355 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../007-librados2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:28.881 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libnbd0. 2024-09-11T04:20:28.908 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-11T04:20:28.947 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-11T04:20:29.216 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librbd1. 2024-09-11T04:20:29.232 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../009-librbd1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:29.275 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:29.728 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T04:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:29.794 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:30.080 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rados. 2024-09-11T04:20:30.107 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:30.146 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:30.449 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T04:20:30.476 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:20:30.516 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:30.701 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T04:20:30.717 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:30.759 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:31.071 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T04:20:31.097 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:20:31.137 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:31.465 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T04:20:31.492 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T04:20:31.532 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T04:20:31.843 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T04:20:31.870 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T04:20:31.909 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T04:20:32.163 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T04:20:32.189 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:32.229 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:32.590 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T04:20:32.617 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T04:20:32.656 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T04:20:32.934 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T04:20:32.961 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T04:20:33.000 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T04:20:33.295 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T04:20:33.311 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T04:20:33.345 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T04:20:33.674 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua5.1. 2024-09-11T04:20:33.700 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T04:20:33.740 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T04:20:34.051 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua-any. 2024-09-11T04:20:34.067 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-11T04:20:34.109 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T04:20:34.337 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package zip. 2024-09-11T04:20:34.353 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-11T04:20:34.387 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T04:20:34.665 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package unzip. 2024-09-11T04:20:34.681 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T04:20:34.715 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T04:20:35.118 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package luarocks. 2024-09-11T04:20:35.145 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T04:20:35.185 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T04:20:35.568 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librgw2. 2024-09-11T04:20:35.595 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../026-librgw2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:35.626 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:36.237 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T04:20:36.264 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:36.303 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:36.639 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T04:20:36.667 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T04:20:36.706 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T04:20:37.000 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T04:20:37.027 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:37.059 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:37.480 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-common. 2024-09-11T04:20:37.506 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:37.545 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:39.251 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-base. 2024-09-11T04:20:39.277 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:39.327 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:39.947 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T04:20:39.973 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T04:20:40.013 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T04:20:40.316 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T04:20:40.344 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T04:20:40.382 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T04:20:40.677 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T04:20:40.704 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T04:20:40.735 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T04:20:41.030 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T04:20:41.057 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T04:20:41.096 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T04:20:41.399 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T04:20:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T04:20:41.465 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T04:20:41.768 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T04:20:41.795 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T04:20:41.835 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T04:20:42.121 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-portend. 2024-09-11T04:20:42.148 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-11T04:20:42.189 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T04:20:42.511 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T04:20:42.538 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T04:20:42.582 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T04:20:42.885 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T04:20:42.912 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T04:20:42.951 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T04:20:43.297 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T04:20:43.324 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T04:20:43.363 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T04:20:43.675 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T04:20:43.702 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T04:20:43.741 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T04:20:44.028 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T04:20:44.054 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T04:20:44.094 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T04:20:44.338 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-mako. 2024-09-11T04:20:44.354 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T04:20:44.388 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T04:20:44.691 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T04:20:44.718 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T04:20:44.758 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T04:20:45.044 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T04:20:45.071 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T04:20:45.110 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T04:20:45.421 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webob. 2024-09-11T04:20:45.448 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T04:20:45.488 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T04:20:45.841 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T04:20:45.868 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T04:20:45.910 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T04:20:46.244 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T04:20:46.270 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T04:20:46.310 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T04:20:46.588 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-paste. 2024-09-11T04:20:46.604 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T04:20:46.646 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T04:20:46.949 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T04:20:46.977 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T04:20:47.016 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T04:20:47.269 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T04:20:47.285 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T04:20:47.319 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T04:20:47.630 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T04:20:47.657 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T04:20:47.696 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T04:20:48.016 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T04:20:48.043 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T04:20:48.082 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T04:20:48.454 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T04:20:48.481 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T04:20:48.521 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T04:20:48.823 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-idna. 2024-09-11T04:20:48.839 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T04:20:48.873 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T04:20:49.143 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T04:20:49.172 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T04:20:49.376 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T04:20:49.696 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests. 2024-09-11T04:20:49.723 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T04:20:49.762 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T04:20:50.124 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T04:20:50.151 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T04:20:50.190 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T04:20:50.468 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T04:20:50.495 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:20:50.535 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:50.923 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T04:20:50.950 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:50.989 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:51.259 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T04:20:51.286 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T04:20:51.326 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T04:20:51.612 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T04:20:51.638 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T04:20:51.678 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T04:20:52.040 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T04:20:52.068 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:52.106 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:52.459 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T04:20:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:52.525 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:54.039 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T04:20:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:54.105 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:55.183 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph. 2024-09-11T04:20:55.199 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../067-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:55.241 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:55.510 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T04:20:55.526 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:55.559 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:55.938 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T04:20:55.965 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:56.004 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:56.407 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package cephadm. 2024-09-11T04:20:56.434 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../070-cephadm_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:20:56.474 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:56.785 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T04:20:56.812 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T04:20:56.868 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T04:20:57.189 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T04:20:57.217 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T04:20:57.256 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T04:20:58.252 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-babel. 2024-09-11T04:20:58.280 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T04:20:58.319 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T04:20:58.638 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T04:20:58.665 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T04:20:58.705 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T04:20:59.033 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T04:20:59.060 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:20:59.108 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:20:59.419 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T04:20:59.447 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T04:20:59.486 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T04:20:59.730 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-routes. 2024-09-11T04:20:59.754 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T04:20:59.797 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T04:21:00.117 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T04:21:00.144 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:21:00.175 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:03.004 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T04:21:03.023 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T04:21:03.062 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T04:21:03.516 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T04:21:03.544 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T04:21:03.574 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T04:21:03.936 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T04:21:03.953 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T04:21:04.014 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T04:21:04.322 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T04:21:04.350 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T04:21:04.474 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T04:21:05.383 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T04:21:05.411 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:21:05.449 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:06.469 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T04:21:06.497 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T04:21:06.535 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T04:21:06.813 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T04:21:06.842 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T04:21:06.880 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T04:21:07.159 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T04:21:07.187 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T04:21:07.225 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T04:21:07.511 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T04:21:07.539 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-11T04:21:07.599 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T04:21:07.932 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T04:21:07.960 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T04:21:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T04:21:08.284 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T04:21:08.301 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T04:21:08.334 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T04:21:08.604 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T04:21:08.632 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T04:21:08.670 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T04:21:09.007 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T04:21:09.036 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T04:21:09.073 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T04:21:09.771 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T04:21:09.788 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:21:09.829 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:10.140 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T04:21:10.169 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T04:21:10.206 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T04:21:10.526 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T04:21:10.555 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T04:21:10.593 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T04:21:10.829 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package jq. 2024-09-11T04:21:10.857 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T04:21:10.895 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T04:21:11.148 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package socat. 2024-09-11T04:21:11.165 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T04:21:11.223 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T04:21:11.534 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T04:21:11.551 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T04:21:11.592 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T04:21:11.964 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-test. 2024-09-11T04:21:11.992 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:21:12.030 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:14.793 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T04:21:14.821 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T04:21:14.867 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:15.172 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T04:21:15.188 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:21:15.229 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:15.841 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T04:21:15.869 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T04:21:15.915 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T04:21:16.270 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T04:21:16.298 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T04:21:16.336 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T04:21:16.682 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T04:21:16.711 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T04:21:16.748 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T04:21:17.195 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package pkg-config. 2024-09-11T04:21:17.224 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T04:21:17.261 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T04:21:17.489 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T04:21:17.507 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T04:21:17.547 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T04:21:17.951 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T04:21:17.968 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T04:21:18.000 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T04:21:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T04:21:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T04:21:18.386 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T04:21:18.706 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T04:21:18.736 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T04:21:18.781 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T04:21:19.092 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T04:21:19.110 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T04:21:19.151 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T04:21:19.512 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-py. 2024-09-11T04:21:19.541 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-11T04:21:19.586 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T04:21:19.915 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T04:21:19.945 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T04:21:19.982 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T04:21:20.394 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T04:21:20.420 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T04:21:20.460 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T04:21:20.755 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-toml. 2024-09-11T04:21:20.784 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-11T04:21:20.821 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T04:21:21.133 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T04:21:21.162 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T04:21:21.199 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T04:21:21.511 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T04:21:21.540 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T04:21:21.577 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T04:21:21.838 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T04:21:21.855 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T04:21:21.896 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T04:21:22.675 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package radosgw. 2024-09-11T04:21:22.705 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../117-radosgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:21:22.742 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:23.645 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T04:21:23.674 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T04:21:23.711 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:24.025 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T04:21:24.283 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T04:21:24.409 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T04:21:24.666 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T04:21:24.987 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T04:21:25.401 INFO:teuthology.orchestra.run.smithi002.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T04:21:25.478 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:26.071 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user cephadm....done 2024-09-11T04:21:26.228 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T04:21:26.694 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T04:21:26.929 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T04:21:27.055 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T04:21:27.314 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T04:21:27.591 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T04:21:27.717 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T04:21:28.027 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T04:21:28.370 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T04:21:28.630 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T04:21:28.906 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T04:21:29.174 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T04:21:29.434 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T04:21:29.693 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T04:21:29.978 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:30.220 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T04:21:30.346 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-11T04:21:30.472 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T04:21:30.598 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T04:21:30.724 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T04:21:31.200 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T04:21:31.585 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T04:21:31.615 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T04:21:31.662 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T04:21:31.787 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T04:21:32.147 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T04:21:32.281 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T04:21:32.574 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T04:21:32.842 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T04:21:33.185 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T04:21:33.286 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T04:21:33.954 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T04:21:34.213 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T04:21:34.340 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T04:21:34.466 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T04:21:34.808 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T04:21:35.201 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T04:21:35.435 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T04:21:35.812 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T04:21:36.113 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T04:21:36.465 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T04:21:36.724 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T04:21:36.850 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:37.151 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T04:21:37.461 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T04:21:38.658 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T04:21:38.902 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T04:21:39.011 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T04:21:39.262 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T04:21:39.387 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T04:21:39.514 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T04:21:39.832 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T04:21:40.133 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T04:21:40.401 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T04:21:40.502 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T04:21:40.803 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T04:21:41.062 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T04:21:41.338 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T04:21:41.648 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T04:21:41.907 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T04:21:42.175 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T04:21:42.276 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T04:21:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T04:21:42.678 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T04:21:43.004 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T04:21:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T04:21:43.583 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T04:21:43.684 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T04:21:43.785 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T04:21:43.858 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T04:21:44.120 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T04:21:44.246 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T04:21:44.639 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T04:21:44.907 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T04:21:45.008 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T04:21:45.251 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T04:21:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T04:21:45.661 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T04:21:45.979 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T04:21:46.263 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T04:21:46.389 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T04:21:46.665 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T04:21:46.994 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T04:21:47.270 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T04:21:47.672 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T04:21:47.965 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T04:21:48.317 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T04:21:48.418 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:48.519 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:48.645 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T04:21:48.981 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:49.107 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:49.234 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T04:21:49.519 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:49.645 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:49.771 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:50.000 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T04:21:50.000 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T04:21:50.497 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:50.648 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:50.773 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:50.900 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T04:21:52.162 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:52.287 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:52.413 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:52.539 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:52.657 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:53.150 INFO:teuthology.orchestra.run.smithi002.stdout:Adding group ceph....done 2024-09-11T04:21:53.535 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user ceph....done 2024-09-11T04:21:53.694 INFO:teuthology.orchestra.run.smithi002.stdout:Setting system user ceph properties....done 2024-09-11T04:21:53.710 INFO:teuthology.orchestra.run.smithi002.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T04:21:53.853 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T04:21:54.167 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T04:21:54.650 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:54.751 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:55.223 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T04:21:55.223 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T04:21:55.711 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:56.095 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T04:21:56.571 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:56.815 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T04:21:56.815 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T04:21:57.297 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:57.523 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T04:21:57.523 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T04:21:57.990 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:58.252 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T04:21:58.252 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T04:21:58.709 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:58.826 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:58.977 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:59.197 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T04:21:59.197 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T04:21:59.737 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:21:59.888 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:22:00.014 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:22:00.165 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T04:22:00.534 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T04:22:00.635 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T04:22:00.787 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T04:22:02.447 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T04:22:03.120 INFO:teuthology.orchestra.run.smithi002.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T04:22:03.126 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:03.127 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:22:03.278 DEBUG:teuthology.orchestra.run.smithi002:> dpkg-query -W -f '${Version}' ceph 2024-09-11T04:22:03.297 INFO:teuthology.orchestra.run.smithi002.stdout:19.3.0-4854-g8f219c47-1jammy 2024-09-11T04:22:03.298 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T04:22:03.298 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854-g8f219c47-1jammy is installed. 2024-09-11T04:22:03.300 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T04:22:03.300 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:03.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T04:22:03.353 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T04:22:03.354 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:03.354 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T04:22:03.409 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T04:22:03.464 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T04:22:03.464 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:03.464 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T04:22:03.521 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T04:22:03.581 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T04:22:03.581 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:03.581 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T04:22:03.636 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T04:22:03.696 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T04:22:03.780 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T04:22:03.780 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /var/log/ceph 2024-09-11T04:22:03.793 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T04:22:03.794 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T04:22:03.850 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T04:22:03.850 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T04:22:03.911 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T04:22:03.912 INFO:tasks.ceph:config {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T04:22:03.912 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900128', 'branch': 'wip-hemanth-testing-2024-09-10-0723', 'description': 'rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}}', 'email': 'hyelloji@redhat.com', 'first_in_suite': False, 'job_id': '7900128', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'owner': 'scheduled_hyelloji@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0']], 'seed': 9167, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-hemanth-testing-2024-09-10-0723', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOpSGdH+8trOW0p0FG1PL5/udYbLGkylBP7EU+5bi0y8OzHGBvQknyFiVwRmn+Rphynhnt2pVRmEKHDzLhNgIIw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['rados/test_health_warnings.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_17:16:14', 'tube': 'smithi', 'user': 'hyelloji', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T04:22:03.912 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T04:22:03.953 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T04:22:04.009 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:04.009 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T04:22:04.057 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T04:22:04.057 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-09-11T04:22:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T04:22:04.105 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:22:04.106 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-09-11T04:22:04.106 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:22:04.106 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-11 04:16:36.194359293 +0000 2024-09-11T04:22:04.106 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-11 04:16:35.946361508 +0000 2024-09-11T04:22:04.106 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-11 04:16:35.946361508 +0000 2024-09-11T04:22:04.106 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-09-11T04:22:04.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T04:22:04.159 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-11T04:22:04.160 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-11T04:22:04.160 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000310878 s, 1.6 MB/s 2024-09-11T04:22:04.161 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T04:22:04.210 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-11 04:16:36.438357113 +0000 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-11 04:16:36.434357149 +0000 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-11 04:16:36.434357149 +0000 2024-09-11T04:22:04.257 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-09-11T04:22:04.258 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T04:22:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-11T04:22:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-11T04:22:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000304564 s, 1.7 MB/s 2024-09-11T04:22:04.312 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T04:22:04.362 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-09-11T04:22:04.409 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T04:22:04.409 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:22:04.409 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-09-11T04:22:04.409 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:22:04.410 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-11 04:16:37.174350539 +0000 2024-09-11T04:22:04.410 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-11 04:16:36.930352719 +0000 2024-09-11T04:22:04.410 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-11 04:16:36.930352719 +0000 2024-09-11T04:22:04.410 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-09-11T04:22:04.410 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T04:22:04.465 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-11T04:22:04.465 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-11T04:22:04.465 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000206345 s, 2.5 MB/s 2024-09-11T04:22:04.466 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T04:22:04.515 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 943 Links: 1 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-11 04:16:37.662346181 +0000 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-11 04:16:37.414348396 +0000 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-11 04:16:37.414348396 +0000 2024-09-11T04:22:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-09-11T04:22:04.566 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T04:22:04.623 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-11T04:22:04.624 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-11T04:22:04.624 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000299971 s, 1.7 MB/s 2024-09-11T04:22:04.625 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T04:22:04.674 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:04.674 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-11T04:22:04.674 INFO:tasks.ceph:Generating config... 2024-09-11T04:22:04.675 INFO:tasks.ceph:[osd] osd max object name len = 400 2024-09-11T04:22:04.676 INFO:tasks.ceph:[osd] osd max object namespace len = 64 2024-09-11T04:22:04.676 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-11T04:22:04.676 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T04:22:04.676 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T04:22:04.676 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-11T04:22:04.676 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-11T04:22:04.676 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-11T04:22:04.677 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-11T04:22:04.677 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-11T04:22:04.677 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-11T04:22:04.677 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-11T04:22:04.677 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T04:22:04.677 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T04:22:04.677 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T04:22:04.677 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T04:22:04.677 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T04:22:04.677 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-11T04:22:04.677 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T04:22:04.678 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T04:22:04.750 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T04:22:04.770 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T04:22:04.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T04:22:04.909 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.2')] 2024-09-11T04:22:04.909 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.2', 'mon election default strategy': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-09-11T04:22:04.912 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:04.912 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T04:22:04.957 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.2 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:05.038 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:05.038 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: generated fsid 6afb2c49-4f7c-48d9-9bd6-caa18d82c311 2024-09-11T04:22:05.038 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = quincy 2024-09-11T04:22:05.038 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 0 2024-09-11T04:22:05.038 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 6afb2c49-4f7c-48d9-9bd6-caa18d82c311 2024-09-11T04:22:05.039 INFO:teuthology.orchestra.run.smithi002.stdout:last_changed 2024-09-11T04:22:05.040278+0000 2024-09-11T04:22:05.039 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-09-11T04:22:05.040278+0000 2024-09-11T04:22:05.039 INFO:teuthology.orchestra.run.smithi002.stdout:min_mon_release 17 (quincy) 2024-09-11T04:22:05.039 INFO:teuthology.orchestra.run.smithi002.stdout:election_strategy: 1 2024-09-11T04:22:05.039 INFO:teuthology.orchestra.run.smithi002.stdout:0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-11T04:22:05.039 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-11T04:22:05.040 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T04:22:05.088 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6afb2c49-4f7c-48d9-9bd6-caa18d82c311... 2024-09-11T04:22:05.089 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T04:22:05.164 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2024-09-11T04:22:05.165 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.166 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T04:22:05.167 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = 172.21.15.2 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: mon election default strategy = 1 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 6afb2c49-4f7c-48d9-9bd6-caa18d82c311 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2024-09-11T04:22:05.168 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 600 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock profile = high_recovery_ops 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug trim objects = true 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.169 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2024-09-11T04:22:05.170 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: osd max object name len = 400 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: osd max object namespace len = 64 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore zero block detection = True 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify cached snaps = True 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify missing on start = True 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock override recovery settings = True 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: osd op queue = debug_random 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: osd op queue cut off = debug_random 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-11T04:22:05.171 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2024-09-11T04:22:05.172 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.173 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: mon scrub interval = 300 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T04:22:05.174 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T04:22:05.175 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2024-09-11T04:22:05.175 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T04:22:05.176 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T04:22:05.270 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T04:22:05.270 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:05.270 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T04:22:05.276 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:05.276 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T04:22:05.320 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:22:05.320 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:05.321 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T04:22:05.321 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T04:22:05.384 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:05.385 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:05.433 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T04:22:05.433 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T04:22:05.433 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T04:22:05.520 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T04:22:05.524 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T04:22:05.524 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T04:22:05.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T04:22:05.560 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T04:22:05.571 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T04:22:05.571 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-11T04:22:05.571 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T04:22:05.624 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:05.624 INFO:tasks.ceph:role: osd.0 2024-09-11T04:22:05.624 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:22:05.624 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T04:22:05.684 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:22:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:22:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:22:05.684 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T04:22:05.685 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:22:05.685 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:22:05.685 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:22:05.685 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T04:22:05.685 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:22:05.685 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:22:05.709 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-09-11T04:22:05.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-09-11T04:22:05.711 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T04:22:05.769 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T04:22:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:22:05.823 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T04:22:05.824 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T04:22:05.876 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:05.876 INFO:tasks.ceph:role: osd.1 2024-09-11T04:22:05.877 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:22:05.877 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T04:22:05.932 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:22:05.933 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:22:05.956 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-09-11T04:22:05.959 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-09-11T04:22:05.959 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T04:22:06.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T04:22:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:22:06.069 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T04:22:06.070 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T04:22:06.125 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.126 INFO:tasks.ceph:role: osd.2 2024-09-11T04:22:06.126 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:22:06.126 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:22:06.181 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:22:06.205 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-09-11T04:22:06.206 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-09-11T04:22:06.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T04:22:06.265 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T04:22:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:22:06.318 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T04:22:06.318 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T04:22:06.373 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.373 INFO:tasks.ceph:role: osd.3 2024-09-11T04:22:06.373 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:22:06.373 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-11T04:22:06.429 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T04:22:06.429 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T04:22:06.430 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T04:22:06.453 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-09-11T04:22:06.455 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-09-11T04:22:06.456 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-11T04:22:06.513 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T04:22:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:22:06.566 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T04:22:06.566 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-11T04:22:06.620 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.620 INFO:tasks.ceph:role: osd.4 2024-09-11T04:22:06.620 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-11T04:22:06.676 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.676 INFO:tasks.ceph:role: osd.5 2024-09-11T04:22:06.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-11T04:22:06.733 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.733 INFO:tasks.ceph:role: osd.6 2024-09-11T04:22:06.733 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-11T04:22:06.788 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.788 INFO:tasks.ceph:role: osd.7 2024-09-11T04:22:06.788 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-11T04:22:06.845 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.845 INFO:tasks.ceph:role: osd.8 2024-09-11T04:22:06.845 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-11T04:22:06.901 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T04:22:06.901 INFO:tasks.ceph:role: osd.9 2024-09-11T04:22:06.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:07.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:07.009+0000 7fbc203f77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T04:22:07.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:07.009+0000 7fbc203f77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T04:22:07.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:07.009+0000 7fbc203f77c0 -1 bdev(0x561fc5af3800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:07.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:07.009+0000 7fbc203f77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T04:22:09.316 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T04:22:09.330 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:09.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:09.413+0000 7f588d3427c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T04:22:09.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:09.413+0000 7f588d3427c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T04:22:09.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:09.413+0000 7f588d3427c0 -1 bdev(0x56434af83800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:09.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:09.413+0000 7f588d3427c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T04:22:11.720 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T04:22:11.777 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:11.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:11.856+0000 7f60d82637c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T04:22:11.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:11.856+0000 7f60d82637c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T04:22:11.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:11.860+0000 7f60d82637c0 -1 bdev(0x558b030cb800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:11.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:11.860+0000 7f60d82637c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T04:22:14.164 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T04:22:14.222 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:14.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:14.304+0000 7f553f6ba7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T04:22:14.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:14.304+0000 7f553f6ba7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T04:22:14.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:14.304+0000 7f553f6ba7c0 -1 bdev(0x560d5d38f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:14.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:14.304+0000 7f553f6ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T04:22:16.621 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T04:22:16.678 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:16.760+0000 7f39898307c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-11T04:22:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:16.760+0000 7f39898307c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-11T04:22:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:16.760+0000 7f39898307c0 -1 bdev(0x5616db805800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:16.760+0000 7f39898307c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-11T04:22:19.099 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-11T04:22:19.153 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:19.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:19.236+0000 7f30b17397c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-11T04:22:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:19.236+0000 7f30b17397c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-11T04:22:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:19.236+0000 7f30b17397c0 -1 bdev(0x5631af61f800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:19.236+0000 7f30b17397c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-11T04:22:21.525 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T04:22:21.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:21.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:21.664+0000 7f742f1907c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-11T04:22:21.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:21.664+0000 7f742f1907c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-11T04:22:21.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:21.664+0000 7f742f1907c0 -1 bdev(0x56056afcd800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:21.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:21.664+0000 7f742f1907c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-11T04:22:23.964 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-11T04:22:24.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:24.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:24.100+0000 7fb79a78d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-11T04:22:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:24.100+0000 7fb79a78d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-11T04:22:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:24.100+0000 7fb79a78d7c0 -1 bdev(0x55c26baeb800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:24.100+0000 7fb79a78d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-11T04:22:26.391 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-11T04:22:26.446 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:26.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:26.528+0000 7fbf2e9ad7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-11T04:22:26.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:26.528+0000 7fbf2e9ad7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-11T04:22:26.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:26.528+0000 7fbf2e9ad7c0 -1 bdev(0x558883a43800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:26.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:26.528+0000 7fbf2e9ad7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-11T04:22:28.831 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-11T04:22:28.890 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:28.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:28.972+0000 7f7c434d07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-11T04:22:28.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:28.972+0000 7f7c434d07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-11T04:22:28.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:28.972+0000 7f7c434d07c0 -1 bdev(0x557b25fd7800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-11T04:22:28.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-09-11T04:22:28.972+0000 7f7c434d07c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-11T04:22:31.279 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-11T04:22:31.338 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T04:22:31.338 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.338 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T04:22:31.394 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.394 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T04:22:31.448 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.448 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T04:22:31.503 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.503 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T04:22:31.561 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T04:22:31.616 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.616 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-11T04:22:31.672 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.672 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-11T04:22:31.728 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.728 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-11T04:22:31.784 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-11T04:22:31.840 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.840 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-11T04:22:31.897 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.897 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-11T04:22:31.952 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:31.952 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T04:22:32.001 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T04:22:32.001 DEBUG:teuthology.orchestra.run.smithi002:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.x] 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDtGuFmIpwnHxAA5NR/aCugoMY+Le5k1O+0Aw== 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.0] 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDvGuFmyzWzABAALZfU3DCd7C47qzD6BGrwWA== 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.1] 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDxGuFmDkC/GBAANOCnh14d+pxehsy4/nvavA== 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.2] 2024-09-11T04:22:32.055 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDzGuFmF0ZiMxAAlFrPXpetckHJm+9vOJbP6g== 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.3] 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD2GuFmnVxJEhAAkc6IIu02I42oCLRfxTrm8g== 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.4] 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD4GuFmaC16LRAABH0jq9wLgALAf1eS9ZvWcQ== 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.5] 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD7GuFm91lADhAA96sCbYQWB0B/WZiRjgjJqA== 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.6] 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQD9GuFm7GzLJxAASMnrq9GOb3JAslaVdLDnIA== 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.7] 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQAAG+FmWD9LBhAAL1WXa7HHKunZT7y4hZ8KOw== 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.8] 2024-09-11T04:22:32.056 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQACG+Fm1UulHxAAz2t3sQQNe1LLnUSBGnrTgQ== 2024-09-11T04:22:32.057 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.9] 2024-09-11T04:22:32.057 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQAEG+Fm9AkvOhAA5kN1YALjk7Lg0IJ+pJtUjQ== 2024-09-11T04:22:32.057 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-09-11T04:22:32.057 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDtGuFmYGWMIRAAw5OMDnTVeehEUhhTSHGHsg== 2024-09-11T04:22:32.058 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T04:22:32.148 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.277 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.362 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.446 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.530 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.618 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.752 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.842 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T04:22:32.926 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T04:22:33.016 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T04:22:33.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T04:22:33.072 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T04:22:33.168 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T04:22:33.179 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T04:22:33.228 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T04:22:33.229 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T04:22:33.229 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T04:22:33.273 INFO:tasks.ceph.mon.a:Started 2024-09-11T04:22:33.274 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph config set mgr mgr_pool false --force 2024-09-11T04:22:33.525 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T04:22:33.525 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T04:22:33.525 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T04:22:33.527 INFO:tasks.ceph.mgr.x:Started 2024-09-11T04:22:33.527 DEBUG:tasks.ceph:set 0 configs 2024-09-11T04:22:33.528 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T04:22:33.528 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T04:22:33.720 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-09-11T04:22:33.738 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T04:22:33.738 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T04:22:33.738 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:33.738 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T04:22:33.749 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:33.749 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T04:22:33.807 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:33.807 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T04:22:33.863 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:33.863 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T04:22:33.920 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:33.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-11T04:22:33.977 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:33.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-11T04:22:34.032 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:34.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-11T04:22:34.088 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:34.088 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-11T04:22:34.144 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:34.144 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-11T04:22:34.199 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:22:34.199 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-11T04:22:34.255 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 86da9381-7ebf-4d49-b779-97f9152b15d9 0 2024-09-11T04:22:34.504 INFO:teuthology.orchestra.run.smithi002.stdout:0 2024-09-11T04:22:34.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new a81b2204-71b5-4a37-b69f-19dbd0999d07 1 2024-09-11T04:22:34.722 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-09-11T04:22:34.739 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new df2297b4-4b3c-47a7-b628-11befe5e0a13 2 2024-09-11T04:22:34.940 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-09-11T04:22:34.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 91dd038e-a387-4088-93b6-1693e5c5e35e 3 2024-09-11T04:22:35.162 INFO:teuthology.orchestra.run.smithi002.stdout:3 2024-09-11T04:22:35.178 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 8bcc434f-9122-4417-81aa-3cb6770d7970 4 2024-09-11T04:22:35.374 INFO:teuthology.orchestra.run.smithi002.stdout:4 2024-09-11T04:22:35.390 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d 5 2024-09-11T04:22:35.586 INFO:teuthology.orchestra.run.smithi002.stdout:5 2024-09-11T04:22:35.602 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 4089cacf-00bd-46e6-90c0-c99186a7b8bb 6 2024-09-11T04:22:35.796 INFO:teuthology.orchestra.run.smithi002.stdout:6 2024-09-11T04:22:35.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624 7 2024-09-11T04:22:36.009 INFO:teuthology.orchestra.run.smithi002.stdout:7 2024-09-11T04:22:36.025 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new dd8535c9-7f50-4b76-8ddb-92d4a777121a 8 2024-09-11T04:22:36.283 INFO:teuthology.orchestra.run.smithi002.stdout:8 2024-09-11T04:22:36.303 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0 9 2024-09-11T04:22:36.495 INFO:teuthology.orchestra.run.smithi002.stdout:9 2024-09-11T04:22:36.514 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T04:22:36.514 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T04:22:36.517 INFO:tasks.ceph.osd.0:Started 2024-09-11T04:22:36.517 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T04:22:36.517 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T04:22:36.519 INFO:tasks.ceph.osd.1:Started 2024-09-11T04:22:36.519 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T04:22:36.519 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T04:22:36.521 INFO:tasks.ceph.osd.2:Started 2024-09-11T04:22:36.521 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T04:22:36.521 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T04:22:36.524 INFO:tasks.ceph.osd.3:Started 2024-09-11T04:22:36.524 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T04:22:36.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T04:22:36.525 INFO:tasks.ceph.mgr.x.smithi002.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T04:22:36.525 INFO:tasks.ceph.mgr.x.smithi002.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T04:22:36.525 INFO:tasks.ceph.mgr.x.smithi002.stderr: from numpy import show_config as show_numpy_config 2024-09-11T04:22:36.527 INFO:tasks.ceph.osd.4:Started 2024-09-11T04:22:36.527 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-11T04:22:36.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-11T04:22:36.529 INFO:tasks.ceph.osd.5:Started 2024-09-11T04:22:36.529 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-11T04:22:36.529 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-11T04:22:36.531 INFO:tasks.ceph.osd.6:Started 2024-09-11T04:22:36.531 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-11T04:22:36.531 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-11T04:22:36.533 INFO:tasks.ceph.osd.7:Started 2024-09-11T04:22:36.533 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-11T04:22:36.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-11T04:22:36.535 INFO:tasks.ceph.osd.8:Started 2024-09-11T04:22:36.535 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-11T04:22:36.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-11T04:22:36.537 INFO:tasks.ceph.osd.9:Started 2024-09-11T04:22:36.538 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:22:36.782 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:36.782 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":12,"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","created":"2024-09-11T04:22:33.407610+0000","modified":"2024-09-11T04:22:36.496762+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T04:22:36.496762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"86da9381-7ebf-4d49-b779-97f9152b15d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a81b2204-71b5-4a37-b69f-19dbd0999d07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df2297b4-4b3c-47a7-b628-11befe5e0a13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"91dd038e-a387-4088-93b6-1693e5c5e35e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8bcc434f-9122-4417-81aa-3cb6770d7970","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4089cacf-00bd-46e6-90c0-c99186a7b8bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"dd8535c9-7f50-4b76-8ddb-92d4a777121a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:22:36.797 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T04:22:36.798 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T04:22:37.761 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:22:37.760+0000 7f11d36027c0 -1 Falling back to public interface 2024-09-11T04:22:37.802 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T04:22:37.978 INFO:teuthology.misc.health.smithi002.stdout: 2024-09-11T04:22:37.978 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":12,"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","created":"2024-09-11T04:22:33.407610+0000","modified":"2024-09-11T04:22:36.496762+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T04:22:36.496762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"86da9381-7ebf-4d49-b779-97f9152b15d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a81b2204-71b5-4a37-b69f-19dbd0999d07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"df2297b4-4b3c-47a7-b628-11befe5e0a13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"91dd038e-a387-4088-93b6-1693e5c5e35e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8bcc434f-9122-4417-81aa-3cb6770d7970","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4089cacf-00bd-46e6-90c0-c99186a7b8bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"dd8535c9-7f50-4b76-8ddb-92d4a777121a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:22:37.993 DEBUG:teuthology.misc:0 of 10 OSDs are up 2024-09-11T04:22:38.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:22:38.020+0000 7f9cfa9517c0 -1 Falling back to public interface 2024-09-11T04:22:38.048 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-09-11T04:22:38.048+0000 7f18baefd7c0 -1 Falling back to public interface 2024-09-11T04:22:38.198 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-09-11T04:22:38.196+0000 7fcd9b3d87c0 -1 Falling back to public interface 2024-09-11T04:22:38.221 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-09-11T04:22:38.220+0000 7f90152af7c0 -1 Falling back to public interface 2024-09-11T04:22:38.272 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:22:38.272+0000 7f25d0aea7c0 -1 Falling back to public interface 2024-09-11T04:22:38.465 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-09-11T04:22:38.464+0000 7fd9a07a77c0 -1 Falling back to public interface 2024-09-11T04:22:38.553 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-09-11T04:22:38.552+0000 7f0e93a167c0 -1 Falling back to public interface 2024-09-11T04:22:38.553 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-09-11T04:22:38.552+0000 7fea9e3497c0 -1 Falling back to public interface 2024-09-11T04:22:38.782 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:22:38.780+0000 7f417736f7c0 -1 Falling back to public interface 2024-09-11T04:22:39.542 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-09-11T04:22:39.540+0000 7f123ac88640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T04:22:41.046 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:22:41.044+0000 7f11d36027c0 -1 osd.0 0 log_to_monitors true 2024-09-11T04:22:41.307 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:22:41.304+0000 7f9cfa9517c0 -1 osd.1 0 log_to_monitors true 2024-09-11T04:22:41.359 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-09-11T04:22:41.356+0000 7f18baefd7c0 -1 osd.5 0 log_to_monitors true 2024-09-11T04:22:41.537 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-09-11T04:22:41.536+0000 7f90152af7c0 -1 osd.4 0 log_to_monitors true 2024-09-11T04:22:41.579 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:22:41.576+0000 7f25d0aea7c0 -1 osd.6 0 log_to_monitors true 2024-09-11T04:22:41.993 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-09-11T04:22:41.992+0000 7fcd9b3d87c0 -1 osd.2 0 log_to_monitors true 2024-09-11T04:22:42.031 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:22:42.028+0000 7f417736f7c0 -1 osd.8 0 log_to_monitors true 2024-09-11T04:22:42.111 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-09-11T04:22:42.108+0000 7f0e93a167c0 -1 osd.3 0 log_to_monitors true 2024-09-11T04:22:42.225 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-09-11T04:22:42.224+0000 7fd9a07a77c0 -1 osd.7 0 log_to_monitors true 2024-09-11T04:22:42.228 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-09-11T04:22:42.228+0000 7fea9e3497c0 -1 osd.9 0 log_to_monitors true 2024-09-11T04:22:43.388 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-09-11T04:22:43.388+0000 7f18b76b8640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T04:22:43.390 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:22:43.388+0000 7f9cf68fa640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T04:22:43.408 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:22:43.404+0000 7f11cfdbd640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T04:22:43.409 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-09-11T04:22:43.408+0000 7f18aecd8640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:43.416 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:22:43.416+0000 7f9cee72c640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:43.441 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:22:43.440+0000 7f11c73dd640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:43.572 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:22:43.572+0000 7f4173318640 -1 osd.8 0 waiting for initial osdmap 2024-09-11T04:22:43.573 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-09-11T04:22:43.572+0000 7f0e8f9bf640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T04:22:43.616 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:22:43.616+0000 7f416f152640 -1 osd.8 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:43.616 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-09-11T04:22:43.616+0000 7f0e8b7f9640 -1 osd.3 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:44.447 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-09-11T04:22:44.444+0000 7fcd97381640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T04:22:44.499 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-09-11T04:22:44.496+0000 7fcd8f1b3640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:44.934 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-09-11T04:22:44.932+0000 7fd99cf62640 -1 osd.7 0 waiting for initial osdmap 2024-09-11T04:22:44.960 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-09-11T04:22:44.960+0000 7fd994582640 -1 osd.7 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:44.962 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:22:44.960+0000 7f25cca93640 -1 osd.6 0 waiting for initial osdmap 2024-09-11T04:22:44.963 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-09-11T04:22:44.964+0000 7fea9a2f2640 -1 osd.9 0 waiting for initial osdmap 2024-09-11T04:22:44.972 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-09-11T04:22:44.972+0000 7f9011258640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T04:22:44.999 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T04:22:45.000 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-09-11T04:22:45.000+0000 7f900908a640 -1 osd.4 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:45.001 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-09-11T04:22:45.000+0000 7fea92124640 -1 osd.9 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:45.001 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:22:45.000+0000 7f25c48c5640 -1 osd.6 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:22:45.297 INFO:teuthology.misc.health.smithi002.stdout: 2024-09-11T04:22:45.297 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":16,"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","created":"2024-09-11T04:22:33.407610+0000","modified":"2024-09-11T04:22:44.576942+0000","last_up_change":"2024-09-11T04:22:44.576942+0000","last_in_change":"2024-09-11T04:22:36.496762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"86da9381-7ebf-4d49-b779-97f9152b15d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6801","nonce":3374428017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6803","nonce":3374428017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6807","nonce":3374428017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6805","nonce":3374428017}]},"public_addr":"172.21.15.2:6801/3374428017","cluster_addr":"172.21.15.2:6803/3374428017","heartbeat_back_addr":"172.21.15.2:6807/3374428017","heartbeat_front_addr":"172.21.15.2:6805/3374428017","state":["exists","up"]},{"osd":1,"uuid":"a81b2204-71b5-4a37-b69f-19dbd0999d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6809","nonce":3277671950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6811","nonce":3277671950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6815","nonce":3277671950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6813","nonce":3277671950}]},"public_addr":"172.21.15.2:6809/3277671950","cluster_addr":"172.21.15.2:6811/3277671950","heartbeat_back_addr":"172.21.15.2:6815/3277671950","heartbeat_front_addr":"172.21.15.2:6813/3277671950","state":["exists","up"]},{"osd":2,"uuid":"df2297b4-4b3c-47a7-b628-11befe5e0a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6825","nonce":151425532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6827","nonce":151425532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6831","nonce":151425532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6829","nonce":151425532}]},"public_addr":"172.21.15.2:6825/151425532","cluster_addr":"172.21.15.2:6827/151425532","heartbeat_back_addr":"172.21.15.2:6831/151425532","heartbeat_front_addr":"172.21.15.2:6829/151425532","state":["exists","up"]},{"osd":3,"uuid":"91dd038e-a387-4088-93b6-1693e5c5e35e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6861","nonce":3860158544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6863","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6865","nonce":3860158544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6871","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6873","nonce":3860158544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6869","nonce":3860158544}]},"public_addr":"172.21.15.2:6861/3860158544","cluster_addr":"172.21.15.2:6865/3860158544","heartbeat_back_addr":"172.21.15.2:6873/3860158544","heartbeat_front_addr":"172.21.15.2:6869/3860158544","state":["exists","up"]},{"osd":4,"uuid":"8bcc434f-9122-4417-81aa-3cb6770d7970","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6817","nonce":512998079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6819","nonce":512998079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6823","nonce":512998079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6821","nonce":512998079}]},"public_addr":"172.21.15.2:6817/512998079","cluster_addr":"172.21.15.2:6819/512998079","heartbeat_back_addr":"172.21.15.2:6823/512998079","heartbeat_front_addr":"172.21.15.2:6821/512998079","state":["exists","up"]},{"osd":6,"uuid":"4089cacf-00bd-46e6-90c0-c99186a7b8bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"dd8535c9-7f50-4b76-8ddb-92d4a777121a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6875","nonce":2913413062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6876","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6877","nonce":2913413062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6880","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6881","nonce":2913413062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6878","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6879","nonce":2913413062}]},"public_addr":"172.21.15.2:6875/2913413062","cluster_addr":"172.21.15.2:6877/2913413062","heartbeat_back_addr":"172.21.15.2:6881/2913413062","heartbeat_front_addr":"172.21.15.2:6879/2913413062","state":["exists","up"]},{"osd":9,"uuid":"3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.054123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.288392+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.314418+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:22:45.315 DEBUG:teuthology.misc:6 of 10 OSDs are up 2024-09-11T04:22:52.322 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T04:22:52.635 INFO:teuthology.misc.health.smithi002.stdout: 2024-09-11T04:22:52.636 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":18,"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","created":"2024-09-11T04:22:33.407610+0000","modified":"2024-09-11T04:22:46.600413+0000","last_up_change":"2024-09-11T04:22:45.590241+0000","last_in_change":"2024-09-11T04:22:36.496762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"86da9381-7ebf-4d49-b779-97f9152b15d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6801","nonce":3374428017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6803","nonce":3374428017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6807","nonce":3374428017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6805","nonce":3374428017}]},"public_addr":"172.21.15.2:6801/3374428017","cluster_addr":"172.21.15.2:6803/3374428017","heartbeat_back_addr":"172.21.15.2:6807/3374428017","heartbeat_front_addr":"172.21.15.2:6805/3374428017","state":["exists","up"]},{"osd":1,"uuid":"a81b2204-71b5-4a37-b69f-19dbd0999d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6809","nonce":3277671950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6811","nonce":3277671950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6815","nonce":3277671950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6813","nonce":3277671950}]},"public_addr":"172.21.15.2:6809/3277671950","cluster_addr":"172.21.15.2:6811/3277671950","heartbeat_back_addr":"172.21.15.2:6815/3277671950","heartbeat_front_addr":"172.21.15.2:6813/3277671950","state":["exists","up"]},{"osd":2,"uuid":"df2297b4-4b3c-47a7-b628-11befe5e0a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6825","nonce":151425532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6827","nonce":151425532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6831","nonce":151425532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6829","nonce":151425532}]},"public_addr":"172.21.15.2:6825/151425532","cluster_addr":"172.21.15.2:6827/151425532","heartbeat_back_addr":"172.21.15.2:6831/151425532","heartbeat_front_addr":"172.21.15.2:6829/151425532","state":["exists","up"]},{"osd":3,"uuid":"91dd038e-a387-4088-93b6-1693e5c5e35e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6861","nonce":3860158544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6863","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6865","nonce":3860158544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6871","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6873","nonce":3860158544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6869","nonce":3860158544}]},"public_addr":"172.21.15.2:6861/3860158544","cluster_addr":"172.21.15.2:6865/3860158544","heartbeat_back_addr":"172.21.15.2:6873/3860158544","heartbeat_front_addr":"172.21.15.2:6869/3860158544","state":["exists","up"]},{"osd":4,"uuid":"8bcc434f-9122-4417-81aa-3cb6770d7970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6833","nonce":2837961144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6835","nonce":2837961144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6839","nonce":2837961144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6837","nonce":2837961144}]},"public_addr":"172.21.15.2:6833/2837961144","cluster_addr":"172.21.15.2:6835/2837961144","heartbeat_back_addr":"172.21.15.2:6839/2837961144","heartbeat_front_addr":"172.21.15.2:6837/2837961144","state":["exists","up"]},{"osd":5,"uuid":"ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6817","nonce":512998079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6819","nonce":512998079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6823","nonce":512998079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6821","nonce":512998079}]},"public_addr":"172.21.15.2:6817/512998079","cluster_addr":"172.21.15.2:6819/512998079","heartbeat_back_addr":"172.21.15.2:6823/512998079","heartbeat_front_addr":"172.21.15.2:6821/512998079","state":["exists","up"]},{"osd":6,"uuid":"4089cacf-00bd-46e6-90c0-c99186a7b8bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6841","nonce":1771532811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6843","nonce":1771532811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6847","nonce":1771532811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6845","nonce":1771532811}]},"public_addr":"172.21.15.2:6841/1771532811","cluster_addr":"172.21.15.2:6843/1771532811","heartbeat_back_addr":"172.21.15.2:6847/1771532811","heartbeat_front_addr":"172.21.15.2:6845/1771532811","state":["exists","up"]},{"osd":7,"uuid":"3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6849","nonce":1091593676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6851","nonce":1091593676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6854","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6855","nonce":1091593676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6852","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6853","nonce":1091593676}]},"public_addr":"172.21.15.2:6849/1091593676","cluster_addr":"172.21.15.2:6851/1091593676","heartbeat_back_addr":"172.21.15.2:6855/1091593676","heartbeat_front_addr":"172.21.15.2:6853/1091593676","state":["exists","up"]},{"osd":8,"uuid":"dd8535c9-7f50-4b76-8ddb-92d4a777121a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6875","nonce":2913413062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6876","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6877","nonce":2913413062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6880","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6881","nonce":2913413062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6878","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6879","nonce":2913413062}]},"public_addr":"172.21.15.2:6875/2913413062","cluster_addr":"172.21.15.2:6877/2913413062","heartbeat_back_addr":"172.21.15.2:6881/2913413062","heartbeat_front_addr":"172.21.15.2:6879/2913413062","state":["exists","up"]},{"osd":9,"uuid":"3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6860","nonce":3427009172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6862","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6864","nonce":3427009172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6870","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6872","nonce":3427009172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6868","nonce":3427009172}]},"public_addr":"172.21.15.2:6860/3427009172","cluster_addr":"172.21.15.2:6864/3427009172","heartbeat_back_addr":"172.21.15.2:6872/3427009172","heartbeat_front_addr":"172.21.15.2:6868/3427009172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.054123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.288392+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.952572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.123353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.494506+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.314418+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.614299+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.185779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.013845+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.252921+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:22:52.654 DEBUG:teuthology.misc:10 of 10 OSDs are up 2024-09-11T04:22:52.655 INFO:tasks.ceph:Creating RBD pool 2024-09-11T04:22:52.655 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T04:22:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'rbd' created 2024-09-11T04:22:53.642 DEBUG:teuthology.orchestra.run.smithi002:> rbd --cluster ceph pool init rbd 2024-09-11T04:22:56.664 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T04:22:56.665 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T04:22:56.666 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T04:22:56.994 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:57.013 INFO:teuthology.orchestra.run.smithi002.stdout:[{"version":10,"timestamp":"2024-09-11T04:22:44.975356+0000","name":"","changes":[{"name":"osd.4/osd_mclock_max_capacity_iops_ssd","new_value":"8338.734007"}]}] 2024-09-11T04:22:57.013 INFO:tasks.ceph_manager:config epoch is 10 2024-09-11T04:22:57.013 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T04:22:57.013 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T04:22:57.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T04:22:57.377 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:57.394 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6856","nonce":1968895305},{"type":"v1","addr":"172.21.15.2:6857","nonce":1968895305}]},"active_addr":"172.21.15.2:6857/1968895305","active_change":"2024-09-11T04:22:38.535527+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":43238451}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1845665872}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2493845103}]}]} 2024-09-11T04:22:57.398 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T04:22:57.398 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T04:22:57.398 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:22:57.712 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:57.712 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","created":"2024-09-11T04:22:33.407610+0000","modified":"2024-09-11T04:22:56.628797+0000","last_up_change":"2024-09-11T04:22:45.590241+0000","last_in_change":"2024-09-11T04:22:36.496762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T04:22:52.970102+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"86da9381-7ebf-4d49-b779-97f9152b15d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6801","nonce":3374428017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6803","nonce":3374428017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6807","nonce":3374428017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6805","nonce":3374428017}]},"public_addr":"172.21.15.2:6801/3374428017","cluster_addr":"172.21.15.2:6803/3374428017","heartbeat_back_addr":"172.21.15.2:6807/3374428017","heartbeat_front_addr":"172.21.15.2:6805/3374428017","state":["exists","up"]},{"osd":1,"uuid":"a81b2204-71b5-4a37-b69f-19dbd0999d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6809","nonce":3277671950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6811","nonce":3277671950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6815","nonce":3277671950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6813","nonce":3277671950}]},"public_addr":"172.21.15.2:6809/3277671950","cluster_addr":"172.21.15.2:6811/3277671950","heartbeat_back_addr":"172.21.15.2:6815/3277671950","heartbeat_front_addr":"172.21.15.2:6813/3277671950","state":["exists","up"]},{"osd":2,"uuid":"df2297b4-4b3c-47a7-b628-11befe5e0a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6825","nonce":151425532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6827","nonce":151425532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6831","nonce":151425532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6829","nonce":151425532}]},"public_addr":"172.21.15.2:6825/151425532","cluster_addr":"172.21.15.2:6827/151425532","heartbeat_back_addr":"172.21.15.2:6831/151425532","heartbeat_front_addr":"172.21.15.2:6829/151425532","state":["exists","up"]},{"osd":3,"uuid":"91dd038e-a387-4088-93b6-1693e5c5e35e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6861","nonce":3860158544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6863","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6865","nonce":3860158544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6871","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6873","nonce":3860158544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6869","nonce":3860158544}]},"public_addr":"172.21.15.2:6861/3860158544","cluster_addr":"172.21.15.2:6865/3860158544","heartbeat_back_addr":"172.21.15.2:6873/3860158544","heartbeat_front_addr":"172.21.15.2:6869/3860158544","state":["exists","up"]},{"osd":4,"uuid":"8bcc434f-9122-4417-81aa-3cb6770d7970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6833","nonce":2837961144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6835","nonce":2837961144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6839","nonce":2837961144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6837","nonce":2837961144}]},"public_addr":"172.21.15.2:6833/2837961144","cluster_addr":"172.21.15.2:6835/2837961144","heartbeat_back_addr":"172.21.15.2:6839/2837961144","heartbeat_front_addr":"172.21.15.2:6837/2837961144","state":["exists","up"]},{"osd":5,"uuid":"ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6817","nonce":512998079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6819","nonce":512998079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6823","nonce":512998079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6821","nonce":512998079}]},"public_addr":"172.21.15.2:6817/512998079","cluster_addr":"172.21.15.2:6819/512998079","heartbeat_back_addr":"172.21.15.2:6823/512998079","heartbeat_front_addr":"172.21.15.2:6821/512998079","state":["exists","up"]},{"osd":6,"uuid":"4089cacf-00bd-46e6-90c0-c99186a7b8bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6841","nonce":1771532811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6843","nonce":1771532811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6847","nonce":1771532811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6845","nonce":1771532811}]},"public_addr":"172.21.15.2:6841/1771532811","cluster_addr":"172.21.15.2:6843/1771532811","heartbeat_back_addr":"172.21.15.2:6847/1771532811","heartbeat_front_addr":"172.21.15.2:6845/1771532811","state":["exists","up"]},{"osd":7,"uuid":"3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6849","nonce":1091593676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6851","nonce":1091593676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6854","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6855","nonce":1091593676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6852","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6853","nonce":1091593676}]},"public_addr":"172.21.15.2:6849/1091593676","cluster_addr":"172.21.15.2:6851/1091593676","heartbeat_back_addr":"172.21.15.2:6855/1091593676","heartbeat_front_addr":"172.21.15.2:6853/1091593676","state":["exists","up"]},{"osd":8,"uuid":"dd8535c9-7f50-4b76-8ddb-92d4a777121a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6875","nonce":2913413062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6876","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6877","nonce":2913413062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6880","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6881","nonce":2913413062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6878","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6879","nonce":2913413062}]},"public_addr":"172.21.15.2:6875/2913413062","cluster_addr":"172.21.15.2:6877/2913413062","heartbeat_back_addr":"172.21.15.2:6881/2913413062","heartbeat_front_addr":"172.21.15.2:6879/2913413062","state":["exists","up"]},{"osd":9,"uuid":"3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6860","nonce":3427009172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6862","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6864","nonce":3427009172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6870","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6872","nonce":3427009172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6868","nonce":3427009172}]},"public_addr":"172.21.15.2:6860/3427009172","cluster_addr":"172.21.15.2:6864/3427009172","heartbeat_back_addr":"172.21.15.2:6872/3427009172","heartbeat_front_addr":"172.21.15.2:6868/3427009172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.054123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.288392+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.952572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.123353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.494506+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.314418+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.614299+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.185779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.013845+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.252921+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:22:57.731 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T04:22:57.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:22:58.039 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:58.039 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","created":"2024-09-11T04:22:33.407610+0000","modified":"2024-09-11T04:22:56.628797+0000","last_up_change":"2024-09-11T04:22:45.590241+0000","last_in_change":"2024-09-11T04:22:36.496762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T04:22:52.970102+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"86da9381-7ebf-4d49-b779-97f9152b15d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6801","nonce":3374428017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6803","nonce":3374428017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6807","nonce":3374428017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6805","nonce":3374428017}]},"public_addr":"172.21.15.2:6801/3374428017","cluster_addr":"172.21.15.2:6803/3374428017","heartbeat_back_addr":"172.21.15.2:6807/3374428017","heartbeat_front_addr":"172.21.15.2:6805/3374428017","state":["exists","up"]},{"osd":1,"uuid":"a81b2204-71b5-4a37-b69f-19dbd0999d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6809","nonce":3277671950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6811","nonce":3277671950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6815","nonce":3277671950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6813","nonce":3277671950}]},"public_addr":"172.21.15.2:6809/3277671950","cluster_addr":"172.21.15.2:6811/3277671950","heartbeat_back_addr":"172.21.15.2:6815/3277671950","heartbeat_front_addr":"172.21.15.2:6813/3277671950","state":["exists","up"]},{"osd":2,"uuid":"df2297b4-4b3c-47a7-b628-11befe5e0a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6825","nonce":151425532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6827","nonce":151425532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6831","nonce":151425532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6829","nonce":151425532}]},"public_addr":"172.21.15.2:6825/151425532","cluster_addr":"172.21.15.2:6827/151425532","heartbeat_back_addr":"172.21.15.2:6831/151425532","heartbeat_front_addr":"172.21.15.2:6829/151425532","state":["exists","up"]},{"osd":3,"uuid":"91dd038e-a387-4088-93b6-1693e5c5e35e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6861","nonce":3860158544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6863","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6865","nonce":3860158544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6871","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6873","nonce":3860158544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6869","nonce":3860158544}]},"public_addr":"172.21.15.2:6861/3860158544","cluster_addr":"172.21.15.2:6865/3860158544","heartbeat_back_addr":"172.21.15.2:6873/3860158544","heartbeat_front_addr":"172.21.15.2:6869/3860158544","state":["exists","up"]},{"osd":4,"uuid":"8bcc434f-9122-4417-81aa-3cb6770d7970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6833","nonce":2837961144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6835","nonce":2837961144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6839","nonce":2837961144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6837","nonce":2837961144}]},"public_addr":"172.21.15.2:6833/2837961144","cluster_addr":"172.21.15.2:6835/2837961144","heartbeat_back_addr":"172.21.15.2:6839/2837961144","heartbeat_front_addr":"172.21.15.2:6837/2837961144","state":["exists","up"]},{"osd":5,"uuid":"ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6817","nonce":512998079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6819","nonce":512998079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6823","nonce":512998079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6821","nonce":512998079}]},"public_addr":"172.21.15.2:6817/512998079","cluster_addr":"172.21.15.2:6819/512998079","heartbeat_back_addr":"172.21.15.2:6823/512998079","heartbeat_front_addr":"172.21.15.2:6821/512998079","state":["exists","up"]},{"osd":6,"uuid":"4089cacf-00bd-46e6-90c0-c99186a7b8bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6841","nonce":1771532811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6843","nonce":1771532811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6847","nonce":1771532811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6845","nonce":1771532811}]},"public_addr":"172.21.15.2:6841/1771532811","cluster_addr":"172.21.15.2:6843/1771532811","heartbeat_back_addr":"172.21.15.2:6847/1771532811","heartbeat_front_addr":"172.21.15.2:6845/1771532811","state":["exists","up"]},{"osd":7,"uuid":"3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6849","nonce":1091593676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6851","nonce":1091593676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6854","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6855","nonce":1091593676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6852","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6853","nonce":1091593676}]},"public_addr":"172.21.15.2:6849/1091593676","cluster_addr":"172.21.15.2:6851/1091593676","heartbeat_back_addr":"172.21.15.2:6855/1091593676","heartbeat_front_addr":"172.21.15.2:6853/1091593676","state":["exists","up"]},{"osd":8,"uuid":"dd8535c9-7f50-4b76-8ddb-92d4a777121a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6875","nonce":2913413062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6876","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6877","nonce":2913413062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6880","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6881","nonce":2913413062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6878","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6879","nonce":2913413062}]},"public_addr":"172.21.15.2:6875/2913413062","cluster_addr":"172.21.15.2:6877/2913413062","heartbeat_back_addr":"172.21.15.2:6881/2913413062","heartbeat_front_addr":"172.21.15.2:6879/2913413062","state":["exists","up"]},{"osd":9,"uuid":"3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6860","nonce":3427009172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6862","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6864","nonce":3427009172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6870","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6872","nonce":3427009172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6868","nonce":3427009172}]},"public_addr":"172.21.15.2:6860/3427009172","cluster_addr":"172.21.15.2:6864/3427009172","heartbeat_back_addr":"172.21.15.2:6872/3427009172","heartbeat_front_addr":"172.21.15.2:6868/3427009172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.054123+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.288392+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.952572+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.123353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.494506+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.314418+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.614299+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.185779+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.013845+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.252921+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:22:58.058 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T04:22:58.058 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T04:22:58.059 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T04:22:58.059 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T04:22:58.060 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T04:22:58.060 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T04:22:58.060 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-11T04:22:58.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-11T04:22:58.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-11T04:22:58.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-11T04:22:58.338 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-09-11T04:22:58.339 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T04:22:58.340 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-09-11T04:22:58.341 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T04:22:58.354 INFO:teuthology.orchestra.run.smithi002.stdout:73014444038 2024-09-11T04:22:58.355 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-11T04:22:58.357 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-09-11T04:22:58.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T04:22:58.372 INFO:teuthology.orchestra.run.smithi002.stdout:73014444036 2024-09-11T04:22:58.372 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-11T04:22:58.392 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-09-11T04:22:58.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T04:22:58.409 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-09-11T04:22:58.409 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T04:22:58.422 INFO:teuthology.orchestra.run.smithi002.stdout:73014444037 2024-09-11T04:22:58.423 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T04:22:58.444 INFO:teuthology.orchestra.run.smithi002.stdout:73014444036 2024-09-11T04:22:58.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-11T04:22:58.458 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-09-11T04:22:58.458 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-11T04:22:58.964 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-09-11T04:22:58.980 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-09-11T04:22:58.980 INFO:teuthology.orchestra.run.smithi002.stdout:73014444036 2024-09-11T04:22:59.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.5 2024-09-11T04:22:59.003 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.3 2024-09-11T04:22:59.028 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.6 2024-09-11T04:22:59.029 INFO:teuthology.orchestra.run.smithi002.stdout:73014444038 2024-09-11T04:22:59.029 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.034 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-09-11T04:22:59.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.0 2024-09-11T04:22:59.054 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444038 for osd.7 2024-09-11T04:22:59.058 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.068 INFO:teuthology.orchestra.run.smithi002.stdout:73014444037 2024-09-11T04:22:59.073 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-09-11T04:22:59.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444037 got 73014444037 for osd.4 2024-09-11T04:22:59.089 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.2 2024-09-11T04:22:59.092 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.112 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-09-11T04:22:59.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.1 2024-09-11T04:22:59.136 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.173 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-09-11T04:22:59.176 INFO:teuthology.orchestra.run.smithi002.stdout:73014444036 2024-09-11T04:22:59.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.8 2024-09-11T04:22:59.193 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.195 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444036 got 73014444036 for osd.9 2024-09-11T04:22:59.195 DEBUG:teuthology.parallel:result is None 2024-09-11T04:22:59.195 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T04:22:59.195 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:22:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:59.527 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:22:59.544 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-11T04:22:58.544936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":301500,"kb_used_data":2088,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048274500,"statfs":{"total":1073741824000,"available":1073433088000,"internally_reserved":0,"allocated":2138112,"data_stored":738448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.910862"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454835+0000","last_change":"2024-09-11T04:22:57.455050+0000","last_active":"2024-09-11T04:22:57.454835+0000","last_peered":"2024-09-11T04:22:57.454835+0000","last_clean":"2024-09-11T04:22:57.454835+0000","last_became_active":"2024-09-11T04:22:55.157555+0000","last_became_peered":"2024-09-11T04:22:55.157555+0000","last_unstale":"2024-09-11T04:22:57.454835+0000","last_undegraded":"2024-09-11T04:22:57.454835+0000","last_fullsized":"2024-09-11T04:22:57.454835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:04.134162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454761+0000","last_change":"2024-09-11T04:22:57.454952+0000","last_active":"2024-09-11T04:22:57.454761+0000","last_peered":"2024-09-11T04:22:57.454761+0000","last_clean":"2024-09-11T04:22:57.454761+0000","last_became_active":"2024-09-11T04:22:55.153205+0000","last_became_peered":"2024-09-11T04:22:55.153205+0000","last_unstale":"2024-09-11T04:22:57.454761+0000","last_undegraded":"2024-09-11T04:22:57.454761+0000","last_fullsized":"2024-09-11T04:22:57.454761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:09.775887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.146596+0000","last_change":"2024-09-11T04:22:57.146787+0000","last_active":"2024-09-11T04:22:57.146596+0000","last_peered":"2024-09-11T04:22:57.146596+0000","last_clean":"2024-09-11T04:22:57.146596+0000","last_became_active":"2024-09-11T04:22:54.636655+0000","last_became_peered":"2024-09-11T04:22:54.636655+0000","last_unstale":"2024-09-11T04:22:57.146596+0000","last_undegraded":"2024-09-11T04:22:57.146596+0000","last_fullsized":"2024-09-11T04:22:57.146596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:04.134162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454465+0000","last_change":"2024-09-11T04:22:57.454686+0000","last_active":"2024-09-11T04:22:57.454465+0000","last_peered":"2024-09-11T04:22:57.454465+0000","last_clean":"2024-09-11T04:22:57.454465+0000","last_became_active":"2024-09-11T04:22:54.634750+0000","last_became_peered":"2024-09-11T04:22:54.634750+0000","last_unstale":"2024-09-11T04:22:57.454465+0000","last_undegraded":"2024-09-11T04:22:57.454465+0000","last_fullsized":"2024-09-11T04:22:57.454465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:53:05.060718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048406799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"20'1","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.146643+0000","last_change":"2024-09-11T04:22:57.146812+0000","last_active":"2024-09-11T04:22:57.146643+0000","last_peered":"2024-09-11T04:22:57.146643+0000","last_clean":"2024-09-11T04:22:57.146643+0000","last_became_active":"2024-09-11T04:22:54.718669+0000","last_became_peered":"2024-09-11T04:22:54.718669+0000","last_unstale":"2024-09-11T04:22:57.146643+0000","last_undegraded":"2024-09-11T04:22:57.146643+0000","last_fullsized":"2024-09-11T04:22:57.146643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:09:28.403012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048580599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"22'2","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:56.643322+0000","last_change":"2024-09-11T04:22:56.643322+0000","last_active":"2024-09-11T04:22:56.643322+0000","last_peered":"2024-09-11T04:22:56.643322+0000","last_clean":"2024-09-11T04:22:56.643322+0000","last_became_active":"2024-09-11T04:22:55.157330+0000","last_became_peered":"2024-09-11T04:22:55.157330+0000","last_unstale":"2024-09-11T04:22:56.643322+0000","last_undegraded":"2024-09-11T04:22:56.643322+0000","last_fullsized":"2024-09-11T04:22:56.643322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:28:36.464864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454400+0000","last_change":"2024-09-11T04:22:57.454633+0000","last_active":"2024-09-11T04:22:57.454400+0000","last_peered":"2024-09-11T04:22:57.454400+0000","last_clean":"2024-09-11T04:22:57.454400+0000","last_became_active":"2024-09-11T04:22:54.634153+0000","last_became_peered":"2024-09-11T04:22:54.634153+0000","last_unstale":"2024-09-11T04:22:57.454400+0000","last_undegraded":"2024-09-11T04:22:57.454400+0000","last_fullsized":"2024-09-11T04:22:57.454400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:25.223105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.259983+0000","last_change":"2024-09-11T04:22:57.260318+0000","last_active":"2024-09-11T04:22:57.259983+0000","last_peered":"2024-09-11T04:22:57.259983+0000","last_clean":"2024-09-11T04:22:57.259983+0000","last_became_active":"2024-09-11T04:22:54.637241+0000","last_became_peered":"2024-09-11T04:22:54.637241+0000","last_unstale":"2024-09-11T04:22:57.259983+0000","last_undegraded":"2024-09-11T04:22:57.259983+0000","last_fullsized":"2024-09-11T04:22:57.259983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:53:05.060718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075760699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826204,"statfs":{"total":107374182400,"available":107342032896,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":17,"seq":73014444038,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826200,"statfs":{"total":107374182400,"available":107342028800,"internally_reserved":0,"allocated":217088,"data_stored":73860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826204,"statfs":{"total":107374182400,"available":107342032896,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":17,"seq":73014444037,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826204,"statfs":{"total":107374182400,"available":107342032896,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":16,"seq":68719476741,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31400,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826200,"statfs":{"total":107374182400,"available":107342028800,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31400,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826200,"statfs":{"total":107374182400,"available":107342028800,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31404,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826196,"statfs":{"total":107374182400,"available":107342024704,"internally_reserved":0,"allocated":217088,"data_stored":73860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:22:59.545 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:22:59.839 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:22:59.840 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:22:59.856 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-11T04:22:58.544936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":301500,"kb_used_data":2088,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048274500,"statfs":{"total":1073741824000,"available":1073433088000,"internally_reserved":0,"allocated":2138112,"data_stored":738448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.910862"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454835+0000","last_change":"2024-09-11T04:22:57.455050+0000","last_active":"2024-09-11T04:22:57.454835+0000","last_peered":"2024-09-11T04:22:57.454835+0000","last_clean":"2024-09-11T04:22:57.454835+0000","last_became_active":"2024-09-11T04:22:55.157555+0000","last_became_peered":"2024-09-11T04:22:55.157555+0000","last_unstale":"2024-09-11T04:22:57.454835+0000","last_undegraded":"2024-09-11T04:22:57.454835+0000","last_fullsized":"2024-09-11T04:22:57.454835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:04.134162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454761+0000","last_change":"2024-09-11T04:22:57.454952+0000","last_active":"2024-09-11T04:22:57.454761+0000","last_peered":"2024-09-11T04:22:57.454761+0000","last_clean":"2024-09-11T04:22:57.454761+0000","last_became_active":"2024-09-11T04:22:55.153205+0000","last_became_peered":"2024-09-11T04:22:55.153205+0000","last_unstale":"2024-09-11T04:22:57.454761+0000","last_undegraded":"2024-09-11T04:22:57.454761+0000","last_fullsized":"2024-09-11T04:22:57.454761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:09.775887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.146596+0000","last_change":"2024-09-11T04:22:57.146787+0000","last_active":"2024-09-11T04:22:57.146596+0000","last_peered":"2024-09-11T04:22:57.146596+0000","last_clean":"2024-09-11T04:22:57.146596+0000","last_became_active":"2024-09-11T04:22:54.636655+0000","last_became_peered":"2024-09-11T04:22:54.636655+0000","last_unstale":"2024-09-11T04:22:57.146596+0000","last_undegraded":"2024-09-11T04:22:57.146596+0000","last_fullsized":"2024-09-11T04:22:57.146596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:04.134162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454465+0000","last_change":"2024-09-11T04:22:57.454686+0000","last_active":"2024-09-11T04:22:57.454465+0000","last_peered":"2024-09-11T04:22:57.454465+0000","last_clean":"2024-09-11T04:22:57.454465+0000","last_became_active":"2024-09-11T04:22:54.634750+0000","last_became_peered":"2024-09-11T04:22:54.634750+0000","last_unstale":"2024-09-11T04:22:57.454465+0000","last_undegraded":"2024-09-11T04:22:57.454465+0000","last_fullsized":"2024-09-11T04:22:57.454465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:53:05.060718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048406799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"20'1","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.146643+0000","last_change":"2024-09-11T04:22:57.146812+0000","last_active":"2024-09-11T04:22:57.146643+0000","last_peered":"2024-09-11T04:22:57.146643+0000","last_clean":"2024-09-11T04:22:57.146643+0000","last_became_active":"2024-09-11T04:22:54.718669+0000","last_became_peered":"2024-09-11T04:22:54.718669+0000","last_unstale":"2024-09-11T04:22:57.146643+0000","last_undegraded":"2024-09-11T04:22:57.146643+0000","last_fullsized":"2024-09-11T04:22:57.146643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:09:28.403012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048580599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"22'2","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:56.643322+0000","last_change":"2024-09-11T04:22:56.643322+0000","last_active":"2024-09-11T04:22:56.643322+0000","last_peered":"2024-09-11T04:22:56.643322+0000","last_clean":"2024-09-11T04:22:56.643322+0000","last_became_active":"2024-09-11T04:22:55.157330+0000","last_became_peered":"2024-09-11T04:22:55.157330+0000","last_unstale":"2024-09-11T04:22:56.643322+0000","last_undegraded":"2024-09-11T04:22:56.643322+0000","last_fullsized":"2024-09-11T04:22:56.643322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:28:36.464864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.454400+0000","last_change":"2024-09-11T04:22:57.454633+0000","last_active":"2024-09-11T04:22:57.454400+0000","last_peered":"2024-09-11T04:22:57.454400+0000","last_clean":"2024-09-11T04:22:57.454400+0000","last_became_active":"2024-09-11T04:22:54.634153+0000","last_became_peered":"2024-09-11T04:22:54.634153+0000","last_unstale":"2024-09-11T04:22:57.454400+0000","last_undegraded":"2024-09-11T04:22:57.454400+0000","last_fullsized":"2024-09-11T04:22:57.454400+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:25.223105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T04:22:57.259983+0000","last_change":"2024-09-11T04:22:57.260318+0000","last_active":"2024-09-11T04:22:57.259983+0000","last_peered":"2024-09-11T04:22:57.259983+0000","last_clean":"2024-09-11T04:22:57.259983+0000","last_became_active":"2024-09-11T04:22:54.637241+0000","last_became_peered":"2024-09-11T04:22:54.637241+0000","last_unstale":"2024-09-11T04:22:57.259983+0000","last_undegraded":"2024-09-11T04:22:57.259983+0000","last_fullsized":"2024-09-11T04:22:57.259983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:53:05.060718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075760699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":17,"seq":73014444036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826204,"statfs":{"total":107374182400,"available":107342032896,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":17,"seq":73014444038,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31400,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826200,"statfs":{"total":107374182400,"available":107342028800,"internally_reserved":0,"allocated":217088,"data_stored":73860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826204,"statfs":{"total":107374182400,"available":107342032896,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":17,"seq":73014444037,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826204,"statfs":{"total":107374182400,"available":107342032896,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":16,"seq":68719476741,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":16,"seq":68719476741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830364,"statfs":{"total":107374182400,"available":107346292736,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31400,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826200,"statfs":{"total":107374182400,"available":107342028800,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31400,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826200,"statfs":{"total":107374182400,"available":107342028800,"internally_reserved":0,"allocated":212992,"data_stored":73841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31404,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826196,"statfs":{"total":107374182400,"available":107342024704,"internally_reserved":0,"allocated":217088,"data_stored":73860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:22:59.857 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T04:22:59.857 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T04:22:59.857 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T04:22:59.857 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T04:23:00.182 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:23:00.182 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T04:23:00.200 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T04:23:00.200 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T04:23:00.216 INFO:tasks.workunit:Pulling workunits from ref 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:23:00.217 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T04:23:00.217 INFO:tasks.workunit:timeout=3h 2024-09-11T04:23:00.217 INFO:tasks.workunit:cleanup=True 2024-09-11T04:23:00.217 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T04:23:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:23:00.223 INFO:teuthology.orchestra.run.smithi002.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T04:23:00.223 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T04:23:00.268 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T04:23:00.269 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T04:23:00.317 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:23:00.366 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T04:23:49.963 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to '8f219c476cb4f26fc5719c661c0323719a45cb30'. 2024-09-11T04:23:49.963 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-09-11T04:23:49.963 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T04:23:49.963 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T04:23:49.963 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T04:23:49.963 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-09-11T04:23:49.963 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-09-11T04:23:49.964 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at 8f219c476cb Merge branch 'mgr-vol-mod-disable' of https://github.com/rishabh-d-dave/ceph into wip-hemanth-testing-2024-09-10-0723 2024-09-11T04:23:49.969 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T04:23:50.016 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T04:23:50.020 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T04:23:50.020 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T04:23:50.778 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T04:23:50.848 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T04:23:50.898 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T04:23:50.899 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T04:23:50.900 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T04:23:50.944 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T04:23:50.948 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-11T04:23:50.948 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T04:23:51.001 INFO:tasks.workunit:Running workunits matching rados/test_health_warnings.sh on client.0... 2024-09-11T04:23:51.002 INFO:tasks.workunit:Running workunit rados/test_health_warnings.sh... 2024-09-11T04:23:51.002 DEBUG:teuthology.orchestra.run.smithi002:workunit test rados/test_health_warnings.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8f219c476cb4f26fc5719c661c0323719a45cb30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2024-09-11T04:23:51.054 INFO:tasks.workunit.client.0.smithi002.stderr:+ crushtool -o crushmap --build --num_osds 10 host straw 2 rack straw 2 row straw 2 root straw 0 2024-09-11T04:23:51.077 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd setcrushmap -i crushmap 2024-09-11T04:23:52.643 INFO:tasks.workunit.client.0.smithi002.stderr:7 2024-09-11T04:23:52.677 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd tree 2024-09-11T04:23:53.086 INFO:tasks.workunit.client.0.smithi002.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-11T04:23:53.086 INFO:tasks.workunit.client.0.smithi002.stdout:-11 10.00000 root root 2024-09-11T04:23:53.086 INFO:tasks.workunit.client.0.smithi002.stdout: -9 8.00000 row row0 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: -6 4.00000 rack rack0 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: -1 2.00000 host host0 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 0 1.00000 osd.0 up 1.00000 1.00000 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 1 1.00000 osd.1 up 1.00000 1.00000 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: -2 2.00000 host host1 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 2 1.00000 osd.2 up 1.00000 1.00000 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 3 1.00000 osd.3 up 1.00000 1.00000 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: -7 4.00000 rack rack1 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: -3 2.00000 host host2 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 4 1.00000 osd.4 up 1.00000 1.00000 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 5 1.00000 osd.5 up 1.00000 1.00000 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: -4 2.00000 host host3 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 6 1.00000 osd.6 up 1.00000 1.00000 2024-09-11T04:23:53.087 INFO:tasks.workunit.client.0.smithi002.stdout: 7 1.00000 osd.7 up 1.00000 1.00000 2024-09-11T04:23:53.088 INFO:tasks.workunit.client.0.smithi002.stdout:-10 2.00000 row row1 2024-09-11T04:23:53.088 INFO:tasks.workunit.client.0.smithi002.stdout: -8 2.00000 rack rack2 2024-09-11T04:23:53.088 INFO:tasks.workunit.client.0.smithi002.stdout: -5 2.00000 host host4 2024-09-11T04:23:53.088 INFO:tasks.workunit.client.0.smithi002.stdout: 8 1.00000 osd.8 up 1.00000 1.00000 2024-09-11T04:23:53.088 INFO:tasks.workunit.client.0.smithi002.stdout: 9 1.00000 osd.9 up 1.00000 1.00000 2024-09-11T04:23:53.104 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph tell 'osd.*' injectargs --osd_max_markdown_count 1024 --osd_max_markdown_period 1 2024-09-11T04:23:53.266 INFO:tasks.workunit.client.0.smithi002.stdout:osd.0: {} 2024-09-11T04:23:53.296 INFO:tasks.workunit.client.0.smithi002.stdout:osd.1: {} 2024-09-11T04:23:53.328 INFO:tasks.workunit.client.0.smithi002.stdout:osd.2: {} 2024-09-11T04:23:53.367 INFO:tasks.workunit.client.0.smithi002.stdout:osd.3: {} 2024-09-11T04:23:53.400 INFO:tasks.workunit.client.0.smithi002.stdout:osd.4: {} 2024-09-11T04:23:53.435 INFO:tasks.workunit.client.0.smithi002.stdout:osd.5: {} 2024-09-11T04:23:53.470 INFO:tasks.workunit.client.0.smithi002.stdout:osd.6: {} 2024-09-11T04:23:53.502 INFO:tasks.workunit.client.0.smithi002.stdout:osd.7: {} 2024-09-11T04:23:53.539 INFO:tasks.workunit.client.0.smithi002.stdout:osd.8: {} 2024-09-11T04:23:53.572 INFO:tasks.workunit.client.0.smithi002.stdout:osd.9: {} 2024-09-11T04:23:53.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noout 2024-09-11T04:23:55.651 INFO:tasks.workunit.client.0.smithi002.stderr:noout is set 2024-09-11T04:23:55.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_two_osds_same_host_down 2024-09-11T04:23:55.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-09-11T04:23:57.663 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-09-11T04:23:57.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down osd.0 osd.1 2024-09-11T04:23:58.008 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:23:58.002+0000 7fd686cd4640 -1 mon.a@0(leader).osd e29 definitely_dead 0 2024-09-11T04:23:58.784 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:23:58.782+0000 7fd686cd4640 -1 mon.a@0(leader).osd e30 definitely_dead 0 2024-09-11T04:23:58.785 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 is already down. osd.1 is already down. 2024-09-11T04:23:58.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:23:59.344 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-11T04:23:59.344 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-11T04:23:59.344 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 2 osds down 2024-09-11T04:23:59.344 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:23:59.344 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:23:59.345 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-09-11T04:23:59.345 INFO:tasks.workunit.client.0.smithi002.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-11T04:23:59.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:23:59.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 host' 2024-09-11T04:23:59.859 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-11T04:23:59.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 osds' 2024-09-11T04:23:59.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:23:59.943 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:23:59.938+0000 7f9cf710c640 -1 osd.1 31 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:00.369 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-11T04:24:00.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:00.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.0 2024-09-11T04:24:00.680 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:00.674+0000 7f9ce670b640 -1 osd.1 32 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:00.894 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:00.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:00.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.1 2024-09-11T04:24:01.427 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:01.427 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-09-11T04:24:02.688 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:02.686+0000 7f9cee72c640 -1 osd.1 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:03.252 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:24:03.250+0000 7f11c73dd640 -1 osd.0 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:03.686 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-09-11T04:24:03.713 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-09-11T04:24:03.713 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:03.714 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-09-11T04:24:04.205 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_two_osds_same_rack_down 2024-09-11T04:24:04.205 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-09-11T04:24:05.696 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-09-11T04:24:05.722 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down osd.8 osd.9 2024-09-11T04:24:06.046 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:24:06.046+0000 7fd686cd4640 -1 mon.a@0(leader).osd e36 definitely_dead 0 2024-09-11T04:24:06.820 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:24:06.818+0000 7fd686cd4640 -1 mon.a@0(leader).osd e37 definitely_dead 0 2024-09-11T04:24:06.820 INFO:tasks.workunit.client.0.smithi002.stderr:osd.8 is already down. osd.9 is already down. 2024-09-11T04:24:06.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:07.311 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-09-11T04:24:07.310+0000 7fea9ab04640 -1 osd.9 37 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 2 osds down 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout: host host4 (root=root,row=row1,rack=rack2) (2 osds) is down 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_RACK_DOWN: 1 rack (2 osds) down 2024-09-11T04:24:07.384 INFO:tasks.workunit.client.0.smithi002.stdout: rack rack2 (root=root,row=row1) (2 osds) is down 2024-09-11T04:24:07.385 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_ROW_DOWN: 1 row (2 osds) down 2024-09-11T04:24:07.385 INFO:tasks.workunit.client.0.smithi002.stdout: row row1 (root=root) (2 osds) is down 2024-09-11T04:24:07.385 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded 2024-09-11T04:24:07.385 INFO:tasks.workunit.client.0.smithi002.stdout: pg 1.2 is active+undersized+degraded, acting [4] 2024-09-11T04:24:07.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:07.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 host' 2024-09-11T04:24:07.584 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:24:07.582+0000 7f4173b2a640 -1 osd.8 37 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:07.705 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-09-11T04:24:07.706+0000 7fea8a103640 -1 osd.9 38 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:07.706 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:24:07.706+0000 7f4167131640 -1 osd.8 38 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:07.898 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded 2024-09-11T04:24:07.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:07.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 rack' 2024-09-11T04:24:08.398 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down; Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded 2024-09-11T04:24:08.399 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:08.399 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 row' 2024-09-11T04:24:08.894 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-09-11T04:24:08.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:08.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 osds' 2024-09-11T04:24:09.369 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-09-11T04:24:09.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:09.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.8 2024-09-11T04:24:09.918 INFO:tasks.workunit.client.0.smithi002.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-11T04:24:09.919 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:09.919 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.9 2024-09-11T04:24:10.454 INFO:tasks.workunit.client.0.smithi002.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-11T04:24:10.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-09-11T04:24:11.729 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:24:11.725+0000 7f416f152640 -1 osd.8 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:11.730 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-09-11T04:24:11.725+0000 7fea92124640 -1 osd.9 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:12.725 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-09-11T04:24:12.753 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-09-11T04:24:12.753 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:12.753 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-09-11T04:24:13.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_all_but_last_osds_down 2024-09-11T04:24:13.231 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-09-11T04:24:14.745 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-09-11T04:24:14.768 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph osd ls 2024-09-11T04:24:14.769 INFO:tasks.workunit.client.0.smithi002.stderr:++ sed '$d' 2024-09-11T04:24:15.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down 0 1 2 3 4 5 6 7 8 2024-09-11T04:24:15.496 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:24:15.493+0000 7fd686cd4640 -1 mon.a@0(leader).osd e42 definitely_dead 0 2024-09-11T04:24:15.865 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:24:15.865+0000 7fd686cd4640 -1 mon.a@0(leader).osd e43 definitely_dead 0 2024-09-11T04:24:15.865 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 is already down. osd.1 is already down. osd.2 is already down. osd.3 is already down. osd.4 is already down. osd.5 is already down. osd.6 is already down. osd.7 is already down. osd.8 is already down. 2024-09-11T04:24:15.883 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:16.353 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:24:16.353+0000 7f25cca93640 -1 osd.6 43 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 9 osds down 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout: osd.2 (root=root,row=row0,rack=rack0,host=host1) is down 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout: osd.3 (root=root,row=row0,rack=rack0,host=host1) is down 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout: osd.4 (root=root,row=row0,rack=rack1,host=host2) is down 2024-09-11T04:24:16.416 INFO:tasks.workunit.client.0.smithi002.stdout: osd.5 (root=root,row=row0,rack=rack1,host=host2) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: osd.6 (root=root,row=row0,rack=rack1,host=host3) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: osd.7 (root=root,row=row0,rack=rack1,host=host3) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 4 hosts (8 osds) down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: host host1 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: host host2 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: host host3 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_RACK_DOWN: 2 racks (8 osds) down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: rack rack0 (root=root,row=row0) (4 osds) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: rack rack1 (root=root,row=row0) (4 osds) is down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_ROW_DOWN: 1 row (8 osds) down 2024-09-11T04:24:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: row row0 (root=root) (8 osds) is down 2024-09-11T04:24:16.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:16.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 row' 2024-09-11T04:24:16.758 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:24:16.757+0000 7f25bc8a4640 -1 osd.6 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:16.913 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-11T04:24:16.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:16.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 racks' 2024-09-11T04:24:17.394 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-11T04:24:17.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '4 hosts' 2024-09-11T04:24:17.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:17.717 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:17.717+0000 7f9cf68fa640 -1 osd.1 44 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:17.760 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:24:17.757+0000 7f25bc8a4640 -1 osd.6 45 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:17.760 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:17.757+0000 7f9ce670b640 -1 osd.1 45 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:17.849 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-09-11T04:24:17.849+0000 7fcd97b93640 -1 osd.2 45 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:17.879 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-11T04:24:17.879 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:17.879 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '9 osds' 2024-09-11T04:24:18.373 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-09-11T04:24:18.373 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-09-11T04:24:18.764 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:24:18.761+0000 7f4173318640 -1 osd.8 45 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:18.772 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-09-11T04:24:18.769+0000 7f416f152640 -1 osd.8 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:18.772 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-09-11T04:24:18.769+0000 7f25c48c5640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:18.773 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:18.773+0000 7f9cee72c640 -1 osd.1 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:18.774 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-09-11T04:24:18.773+0000 7fcd8f1b3640 -1 osd.2 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:18.811 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-09-11T04:24:18.809+0000 7fd994582640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:19.124 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:24:19.121+0000 7f11c73dd640 -1 osd.0 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:19.201 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-09-11T04:24:19.201+0000 7f0e8b7f9640 -1 osd.3 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:19.775 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-09-11T04:24:19.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-09-11T04:24:19.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:19.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-09-11T04:24:20.325 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-11T04:24:20.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 1 2024-09-11T04:24:21.062 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-09-11T04:24:21.061+0000 7f900908a640 -1 osd.4 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:21.327 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:21.327 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-09-11T04:24:21.469 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-09-11T04:24:21.469+0000 7f18aecd8640 -1 osd.5 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:21.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_two_osds_same_host_down_with_classes 2024-09-11T04:24:21.833 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-09-11T04:24:23.801 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-09-11T04:24:23.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd crush set-device-class ssd osd.0 osd.2 osd.4 osd.6 osd.8 2024-09-11T04:24:25.811 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. osd.2 already set to class ssd. set-device-class item id 2 name 'osd.2' device_class 'ssd': no change. osd.4 already set to class ssd. set-device-class item id 4 name 'osd.4' device_class 'ssd': no change. osd.6 already set to class ssd. set-device-class item id 6 name 'osd.6' device_class 'ssd': no change. osd.8 already set to class ssd. set-device-class item id 8 name 'osd.8' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-11T04:24:25.837 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd crush set-device-class hdd osd.1 osd.3 osd.5 osd.7 osd.9 2024-09-11T04:24:27.823 INFO:tasks.workunit.client.0.smithi002.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. osd.3 already set to class hdd. set-device-class item id 3 name 'osd.3' device_class 'hdd': no change. osd.5 already set to class hdd. set-device-class item id 5 name 'osd.5' device_class 'hdd': no change. osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. osd.9 already set to class hdd. set-device-class item id 9 name 'osd.9' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-11T04:24:27.849 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down osd.0 osd.1 2024-09-11T04:24:28.170 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:24:28.169+0000 7fd686cd4640 -1 mon.a@0(leader).osd e55 definitely_dead 0 2024-09-11T04:24:28.949 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-09-11T04:24:28.945+0000 7fd686cd4640 -1 mon.a@0(leader).osd e56 definitely_dead 0 2024-09-11T04:24:28.949 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 is already down. osd.1 is already down. 2024-09-11T04:24:28.967 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:29.493 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-11T04:24:29.494 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-09-11T04:24:29.494 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 2 osds down 2024-09-11T04:24:29.494 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:29.494 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:29.494 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-09-11T04:24:29.494 INFO:tasks.workunit.client.0.smithi002.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-09-11T04:24:29.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:29.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 host' 2024-09-11T04:24:29.805 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:29.801+0000 7f9cf710c640 -1 osd.1 56 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:29.833 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:29.833+0000 7f9ce670b640 -1 osd.1 57 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:30.008 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-11T04:24:30.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:30.009 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 osds' 2024-09-11T04:24:30.244 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:24:30.241+0000 7f11cfdbd640 -1 osd.0 57 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:30.508 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-09-11T04:24:30.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:30.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.0 2024-09-11T04:24:30.843 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:24:30.841+0000 7f11bf3bc640 -1 osd.0 58 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:30.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:30.841+0000 7f9ce670b640 -1 osd.1 58 osdmap NOUP flag is set, waiting for it to clear 2024-09-11T04:24:31.047 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:31.048 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-09-11T04:24:31.048 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.1 2024-09-11T04:24:31.580 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-09-11T04:24:31.581 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-09-11T04:24:32.852 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-09-11T04:24:32.849+0000 7f11c73dd640 -1 osd.0 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:32.854 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-09-11T04:24:32.849+0000 7f9cee72c640 -1 osd.1 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T04:24:33.851 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-09-11T04:24:33.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-09-11T04:24:33.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-09-11T04:24:33.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-09-11T04:24:34.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ exit 0 2024-09-11T04:24:34.383 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T04:24:34.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T04:24:34.436 INFO:tasks.workunit:Stopping ['rados/test_health_warnings.sh'] on client.0... 2024-09-11T04:24:34.436 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T04:24:34.793 DEBUG:teuthology.parallel:result is None 2024-09-11T04:24:34.793 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T04:24:34.805 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T04:24:34.806 DEBUG:teuthology.orchestra.run.smithi002:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T04:24:34.867 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T04:24:34.868 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T04:24:34.877 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T04:24:34.877 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:35.223 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:35.243 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-11T04:24:34.854057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":13,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":305836,"kb_used_data":6396,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048270164,"statfs":{"total":1073741824000,"available":1073428647936,"internally_reserved":0,"allocated":6549504,"data_stored":3116218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.287805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T04:24:29.838925+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:29.838925+0000","last_peered":"2024-09-11T04:24:29.838925+0000","last_clean":"2024-09-11T04:24:29.838925+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:29.838925+0000","last_undegraded":"2024-09-11T04:24:29.838925+0000","last_fullsized":"2024-09-11T04:24:29.838925+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:30.850394+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:30.850394+0000","last_peered":"2024-09-11T04:24:30.850394+0000","last_clean":"2024-09-11T04:24:30.850394+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:30.850394+0000","last_undegraded":"2024-09-11T04:24:30.850394+0000","last_fullsized":"2024-09-11T04:24:30.850394+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":93,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-09-11T04:24:30.848692+0000","last_change":"2024-09-11T04:24:30.848300+0000","last_active":"2024-09-11T04:24:30.848692+0000","last_peered":"2024-09-11T04:24:30.848692+0000","last_clean":"2024-09-11T04:24:27.826985+0000","last_became_active":"2024-09-11T04:24:30.848300+0000","last_became_peered":"2024-09-11T04:24:30.848300+0000","last_unstale":"2024-09-11T04:24:30.848692+0000","last_undegraded":"2024-09-11T04:24:30.848692+0000","last_fullsized":"2024-09-11T04:24:30.845653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-11T04:24:34.028224+0000","last_change":"2024-09-11T04:24:33.856626+0000","last_active":"2024-09-11T04:24:33.856626+0000","last_peered":"2024-09-11T04:24:28.164534+0000","last_clean":"2024-09-11T04:24:28.164534+0000","last_became_active":"2024-09-11T04:24:20.790226+0000","last_became_peered":"2024-09-11T04:24:20.790226+0000","last_unstale":"2024-09-11T04:24:34.028224+0000","last_undegraded":"2024-09-11T04:24:34.028224+0000","last_fullsized":"2024-09-11T04:24:34.028224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":84,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:24:29.840704+0000","last_change":"2024-09-11T04:24:29.840328+0000","last_active":"2024-09-11T04:24:29.840704+0000","last_peered":"2024-09-11T04:24:29.840704+0000","last_clean":"2024-09-11T04:24:28.165331+0000","last_became_active":"2024-09-11T04:24:29.840328+0000","last_became_peered":"2024-09-11T04:24:29.840328+0000","last_unstale":"2024-09-11T04:24:29.840704+0000","last_undegraded":"2024-09-11T04:24:29.839274+0000","last_fullsized":"2024-09-11T04:24:29.839035+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:33:06.948938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":84,"reported_epoch":58,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:24:30.852998+0000","last_change":"2024-09-11T04:24:30.852351+0000","last_active":"2024-09-11T04:24:30.852998+0000","last_peered":"2024-09-11T04:24:30.852998+0000","last_clean":"2024-09-11T04:24:28.167684+0000","last_became_active":"2024-09-11T04:24:30.852351+0000","last_became_peered":"2024-09-11T04:24:30.852351+0000","last_unstale":"2024-09-11T04:24:30.852998+0000","last_undegraded":"2024-09-11T04:24:30.851355+0000","last_fullsized":"2024-09-11T04:24:30.851271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:43:22.517384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":71,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:31.209293+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:31.209293+0000","last_peered":"2024-09-11T04:24:31.209293+0000","last_clean":"2024-09-11T04:24:31.209293+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:31.209293+0000","last_undegraded":"2024-09-11T04:24:31.209293+0000","last_fullsized":"2024-09-11T04:24:31.209293+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T04:24:33.524789+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:33.524789+0000","last_peered":"2024-09-11T04:24:33.524789+0000","last_clean":"2024-09-11T04:24:33.524789+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:33.524789+0000","last_undegraded":"2024-09-11T04:24:33.524789+0000","last_fullsized":"2024-09-11T04:24:33.524789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":13,"acting":13,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31832,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825768,"statfs":{"total":107374182400,"available":107341586432,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397528,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31808,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825792,"statfs":{"total":107374182400,"available":107341611008,"internally_reserved":0,"allocated":634880,"data_stored":296933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27664,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462936,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27664,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825760,"statfs":{"total":107374182400,"available":107341578240,"internally_reserved":0,"allocated":667648,"data_stored":319954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037784,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825756,"statfs":{"total":107374182400,"available":107341574144,"internally_reserved":0,"allocated":671744,"data_stored":319973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":679936,"data_stored":328430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:35.244 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:35.541 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:35.541 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:35.558 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-11T04:24:34.854057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":13,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":305836,"kb_used_data":6396,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048270164,"statfs":{"total":1073741824000,"available":1073428647936,"internally_reserved":0,"allocated":6549504,"data_stored":3116218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.287805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T04:24:29.838925+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:29.838925+0000","last_peered":"2024-09-11T04:24:29.838925+0000","last_clean":"2024-09-11T04:24:29.838925+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:29.838925+0000","last_undegraded":"2024-09-11T04:24:29.838925+0000","last_fullsized":"2024-09-11T04:24:29.838925+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:30.850394+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:30.850394+0000","last_peered":"2024-09-11T04:24:30.850394+0000","last_clean":"2024-09-11T04:24:30.850394+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:30.850394+0000","last_undegraded":"2024-09-11T04:24:30.850394+0000","last_fullsized":"2024-09-11T04:24:30.850394+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":93,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-09-11T04:24:30.848692+0000","last_change":"2024-09-11T04:24:30.848300+0000","last_active":"2024-09-11T04:24:30.848692+0000","last_peered":"2024-09-11T04:24:30.848692+0000","last_clean":"2024-09-11T04:24:27.826985+0000","last_became_active":"2024-09-11T04:24:30.848300+0000","last_became_peered":"2024-09-11T04:24:30.848300+0000","last_unstale":"2024-09-11T04:24:30.848692+0000","last_undegraded":"2024-09-11T04:24:30.848692+0000","last_fullsized":"2024-09-11T04:24:30.845653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-11T04:24:34.028224+0000","last_change":"2024-09-11T04:24:33.856626+0000","last_active":"2024-09-11T04:24:33.856626+0000","last_peered":"2024-09-11T04:24:28.164534+0000","last_clean":"2024-09-11T04:24:28.164534+0000","last_became_active":"2024-09-11T04:24:20.790226+0000","last_became_peered":"2024-09-11T04:24:20.790226+0000","last_unstale":"2024-09-11T04:24:34.028224+0000","last_undegraded":"2024-09-11T04:24:34.028224+0000","last_fullsized":"2024-09-11T04:24:34.028224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":84,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:24:29.840704+0000","last_change":"2024-09-11T04:24:29.840328+0000","last_active":"2024-09-11T04:24:29.840704+0000","last_peered":"2024-09-11T04:24:29.840704+0000","last_clean":"2024-09-11T04:24:28.165331+0000","last_became_active":"2024-09-11T04:24:29.840328+0000","last_became_peered":"2024-09-11T04:24:29.840328+0000","last_unstale":"2024-09-11T04:24:29.840704+0000","last_undegraded":"2024-09-11T04:24:29.839274+0000","last_fullsized":"2024-09-11T04:24:29.839035+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:33:06.948938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":84,"reported_epoch":58,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:24:30.852998+0000","last_change":"2024-09-11T04:24:30.852351+0000","last_active":"2024-09-11T04:24:30.852998+0000","last_peered":"2024-09-11T04:24:30.852998+0000","last_clean":"2024-09-11T04:24:28.167684+0000","last_became_active":"2024-09-11T04:24:30.852351+0000","last_became_peered":"2024-09-11T04:24:30.852351+0000","last_unstale":"2024-09-11T04:24:30.852998+0000","last_undegraded":"2024-09-11T04:24:30.851355+0000","last_fullsized":"2024-09-11T04:24:30.851271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:43:22.517384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":71,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:31.209293+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:31.209293+0000","last_peered":"2024-09-11T04:24:31.209293+0000","last_clean":"2024-09-11T04:24:31.209293+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:31.209293+0000","last_undegraded":"2024-09-11T04:24:31.209293+0000","last_fullsized":"2024-09-11T04:24:31.209293+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T04:24:33.524789+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:33.524789+0000","last_peered":"2024-09-11T04:24:33.524789+0000","last_clean":"2024-09-11T04:24:33.524789+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:33.524789+0000","last_undegraded":"2024-09-11T04:24:33.524789+0000","last_fullsized":"2024-09-11T04:24:33.524789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":13,"acting":13,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31832,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825768,"statfs":{"total":107374182400,"available":107341586432,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397528,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31808,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825792,"statfs":{"total":107374182400,"available":107341611008,"internally_reserved":0,"allocated":634880,"data_stored":296933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27664,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462936,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27664,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825760,"statfs":{"total":107374182400,"available":107341578240,"internally_reserved":0,"allocated":667648,"data_stored":319954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037784,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825756,"statfs":{"total":107374182400,"available":107341574144,"internally_reserved":0,"allocated":671744,"data_stored":319973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":679936,"data_stored":328430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:35.559 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:35.866 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:35.866 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:35.885 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-11T04:24:34.854057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":13,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":305836,"kb_used_data":6396,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048270164,"statfs":{"total":1073741824000,"available":1073428647936,"internally_reserved":0,"allocated":6549504,"data_stored":3116218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.287805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-11T04:24:29.838925+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:29.838925+0000","last_peered":"2024-09-11T04:24:29.838925+0000","last_clean":"2024-09-11T04:24:29.838925+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:29.838925+0000","last_undegraded":"2024-09-11T04:24:29.838925+0000","last_fullsized":"2024-09-11T04:24:29.838925+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:30.850394+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:30.850394+0000","last_peered":"2024-09-11T04:24:30.850394+0000","last_clean":"2024-09-11T04:24:30.850394+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:30.850394+0000","last_undegraded":"2024-09-11T04:24:30.850394+0000","last_fullsized":"2024-09-11T04:24:30.850394+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":93,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-09-11T04:24:30.848692+0000","last_change":"2024-09-11T04:24:30.848300+0000","last_active":"2024-09-11T04:24:30.848692+0000","last_peered":"2024-09-11T04:24:30.848692+0000","last_clean":"2024-09-11T04:24:27.826985+0000","last_became_active":"2024-09-11T04:24:30.848300+0000","last_became_peered":"2024-09-11T04:24:30.848300+0000","last_unstale":"2024-09-11T04:24:30.848692+0000","last_undegraded":"2024-09-11T04:24:30.848692+0000","last_fullsized":"2024-09-11T04:24:30.845653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-11T04:24:34.028224+0000","last_change":"2024-09-11T04:24:33.856626+0000","last_active":"2024-09-11T04:24:33.856626+0000","last_peered":"2024-09-11T04:24:28.164534+0000","last_clean":"2024-09-11T04:24:28.164534+0000","last_became_active":"2024-09-11T04:24:20.790226+0000","last_became_peered":"2024-09-11T04:24:20.790226+0000","last_unstale":"2024-09-11T04:24:34.028224+0000","last_undegraded":"2024-09-11T04:24:34.028224+0000","last_fullsized":"2024-09-11T04:24:34.028224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":84,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:24:29.840704+0000","last_change":"2024-09-11T04:24:29.840328+0000","last_active":"2024-09-11T04:24:29.840704+0000","last_peered":"2024-09-11T04:24:29.840704+0000","last_clean":"2024-09-11T04:24:28.165331+0000","last_became_active":"2024-09-11T04:24:29.840328+0000","last_became_peered":"2024-09-11T04:24:29.840328+0000","last_unstale":"2024-09-11T04:24:29.840704+0000","last_undegraded":"2024-09-11T04:24:29.839274+0000","last_fullsized":"2024-09-11T04:24:29.839035+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:33:06.948938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":84,"reported_epoch":58,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:24:30.852998+0000","last_change":"2024-09-11T04:24:30.852351+0000","last_active":"2024-09-11T04:24:30.852998+0000","last_peered":"2024-09-11T04:24:30.852998+0000","last_clean":"2024-09-11T04:24:28.167684+0000","last_became_active":"2024-09-11T04:24:30.852351+0000","last_became_peered":"2024-09-11T04:24:30.852351+0000","last_unstale":"2024-09-11T04:24:30.852998+0000","last_undegraded":"2024-09-11T04:24:30.851355+0000","last_fullsized":"2024-09-11T04:24:30.851271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:43:22.517384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":71,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:31.209293+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:31.209293+0000","last_peered":"2024-09-11T04:24:31.209293+0000","last_clean":"2024-09-11T04:24:31.209293+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:31.209293+0000","last_undegraded":"2024-09-11T04:24:31.209293+0000","last_fullsized":"2024-09-11T04:24:31.209293+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T04:24:33.524789+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:33.524789+0000","last_peered":"2024-09-11T04:24:33.524789+0000","last_clean":"2024-09-11T04:24:33.524789+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:33.524789+0000","last_undegraded":"2024-09-11T04:24:33.524789+0000","last_fullsized":"2024-09-11T04:24:33.524789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":13,"acting":13,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31832,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825768,"statfs":{"total":107374182400,"available":107341586432,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397528,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31808,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825792,"statfs":{"total":107374182400,"available":107341611008,"internally_reserved":0,"allocated":634880,"data_stored":296933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27664,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462936,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27664,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829936,"statfs":{"total":107374182400,"available":107345854464,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":643072,"data_stored":304658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825760,"statfs":{"total":107374182400,"available":107341578240,"internally_reserved":0,"allocated":667648,"data_stored":319954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037784,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825756,"statfs":{"total":107374182400,"available":107341574144,"internally_reserved":0,"allocated":671744,"data_stored":319973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":679936,"data_stored":328430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:35.886 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2024-09-11T04:24:35.887 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 93, 'reported_epoch': 58, 'state': 'active+undersized', 'last_fresh': '2024-09-11T04:24:30.848692+0000', 'last_change': '2024-09-11T04:24:30.848300+0000', 'last_active': '2024-09-11T04:24:30.848692+0000', 'last_peered': '2024-09-11T04:24:30.848692+0000', 'last_clean': '2024-09-11T04:24:27.826985+0000', 'last_became_active': '2024-09-11T04:24:30.848300+0000', 'last_became_peered': '2024-09-11T04:24:30.848300+0000', 'last_unstale': '2024-09-11T04:24:30.848692+0000', 'last_undegraded': '2024-09-11T04:24:30.848692+0000', 'last_fullsized': '2024-09-11T04:24:30.845653+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_clean_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.00046493, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9], 'acting': [9], 'avail_no_missing': ['9'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-09-11T04:24:35.887 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2024-09-11T04:24:35.887 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 70, 'reported_epoch': 60, 'state': 'peering', 'last_fresh': '2024-09-11T04:24:34.028224+0000', 'last_change': '2024-09-11T04:24:33.856626+0000', 'last_active': '2024-09-11T04:24:33.856626+0000', 'last_peered': '2024-09-11T04:24:28.164534+0000', 'last_clean': '2024-09-11T04:24:28.164534+0000', 'last_became_active': '2024-09-11T04:24:20.790226+0000', 'last_became_peered': '2024-09-11T04:24:20.790226+0000', 'last_unstale': '2024-09-11T04:24:34.028224+0000', 'last_undegraded': '2024-09-11T04:24:34.028224+0000', 'last_fullsized': '2024-09-11T04:24:34.028224+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_clean_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T04:24:35.887 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2024-09-11T04:24:35.887 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "20'1", 'reported_seq': 84, 'reported_epoch': 57, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-11T04:24:29.840704+0000', 'last_change': '2024-09-11T04:24:29.840328+0000', 'last_active': '2024-09-11T04:24:29.840704+0000', 'last_peered': '2024-09-11T04:24:29.840704+0000', 'last_clean': '2024-09-11T04:24:28.165331+0000', 'last_became_active': '2024-09-11T04:24:29.840328+0000', 'last_became_peered': '2024-09-11T04:24:29.840328+0000', 'last_unstale': '2024-09-11T04:24:29.840704+0000', 'last_undegraded': '2024-09-11T04:24:29.839274+0000', 'last_fullsized': '2024-09-11T04:24:29.839035+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_clean_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-12T13:33:06.948938+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-11T04:24:35.887 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2024-09-11T04:24:35.888 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "22'2", 'reported_seq': 84, 'reported_epoch': 58, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-11T04:24:30.852998+0000', 'last_change': '2024-09-11T04:24:30.852351+0000', 'last_active': '2024-09-11T04:24:30.852998+0000', 'last_peered': '2024-09-11T04:24:30.852998+0000', 'last_clean': '2024-09-11T04:24:28.167684+0000', 'last_became_active': '2024-09-11T04:24:30.852351+0000', 'last_became_peered': '2024-09-11T04:24:30.852351+0000', 'last_unstale': '2024-09-11T04:24:30.852998+0000', 'last_undegraded': '2024-09-11T04:24:30.851355+0000', 'last_fullsized': '2024-09-11T04:24:30.851271+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 50, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_clean_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-12T14:43:22.517384+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001965961, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [{'shards': '4', 'objects': 1}], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-11T04:24:35.888 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:24:36.259 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:36.259 INFO:teuthology.orchestra.run.smithi002.stdout:{"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":122,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":10,"num_up_osds":10,"osd_up_since":1726028673,"num_in_osds":10,"osd_in_since":1726028556,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+undersized+degraded","count":2},{"state_name":"active+undersized","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":313044992,"bytes_avail":1073428779008,"bytes_total":1073741824000,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"btime":"2024-09-11T04:22:33:407386+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-11T04:23:48.556119+0000","services":{}},"progress_events":{}} 2024-09-11T04:24:36.279 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-11T04:24:36.279 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:36.590 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:36.590 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:36.608 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-11T04:24:36.567962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":305988,"kb_used_data":6540,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048270012,"statfs":{"total":1073741824000,"available":1073428492288,"internally_reserved":0,"allocated":6696960,"data_stored":3210538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.724760"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.856625+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:34.856625+0000","last_peered":"2024-09-11T04:24:34.856625+0000","last_clean":"2024-09-11T04:24:34.856625+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:34.856625+0000","last_undegraded":"2024-09-11T04:24:34.856625+0000","last_fullsized":"2024-09-11T04:24:34.856625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:30.850394+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:30.850394+0000","last_peered":"2024-09-11T04:24:30.850394+0000","last_clean":"2024-09-11T04:24:30.850394+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:30.850394+0000","last_undegraded":"2024-09-11T04:24:30.850394+0000","last_fullsized":"2024-09-11T04:24:30.850394+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":93,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-09-11T04:24:30.848692+0000","last_change":"2024-09-11T04:24:30.848300+0000","last_active":"2024-09-11T04:24:30.848692+0000","last_peered":"2024-09-11T04:24:30.848692+0000","last_clean":"2024-09-11T04:24:27.826985+0000","last_became_active":"2024-09-11T04:24:30.848300+0000","last_became_peered":"2024-09-11T04:24:30.848300+0000","last_unstale":"2024-09-11T04:24:30.848692+0000","last_undegraded":"2024-09-11T04:24:30.848692+0000","last_fullsized":"2024-09-11T04:24:30.845653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-11T04:24:34.028224+0000","last_change":"2024-09-11T04:24:33.856626+0000","last_active":"2024-09-11T04:24:33.856626+0000","last_peered":"2024-09-11T04:24:28.164534+0000","last_clean":"2024-09-11T04:24:28.164534+0000","last_became_active":"2024-09-11T04:24:20.790226+0000","last_became_peered":"2024-09-11T04:24:20.790226+0000","last_unstale":"2024-09-11T04:24:34.028224+0000","last_undegraded":"2024-09-11T04:24:34.028224+0000","last_fullsized":"2024-09-11T04:24:34.028224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":98,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859303+0000","last_change":"2024-09-11T04:24:34.859204+0000","last_active":"2024-09-11T04:24:34.859303+0000","last_peered":"2024-09-11T04:24:34.859303+0000","last_clean":"2024-09-11T04:24:34.859303+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:34.859303+0000","last_undegraded":"2024-09-11T04:24:34.859303+0000","last_fullsized":"2024-09-11T04:24:34.859303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:24.877151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":84,"reported_epoch":58,"state":"active+undersized+degraded","last_fresh":"2024-09-11T04:24:30.852998+0000","last_change":"2024-09-11T04:24:30.852351+0000","last_active":"2024-09-11T04:24:30.852998+0000","last_peered":"2024-09-11T04:24:30.852998+0000","last_clean":"2024-09-11T04:24:28.167684+0000","last_became_active":"2024-09-11T04:24:30.852351+0000","last_became_peered":"2024-09-11T04:24:30.852351+0000","last_unstale":"2024-09-11T04:24:30.852998+0000","last_undegraded":"2024-09-11T04:24:30.851355+0000","last_fullsized":"2024-09-11T04:24:30.851271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:43:22.517384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":71,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-11T04:24:31.209293+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:31.209293+0000","last_peered":"2024-09-11T04:24:31.209293+0000","last_clean":"2024-09-11T04:24:31.209293+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:31.209293+0000","last_undegraded":"2024-09-11T04:24:31.209293+0000","last_fullsized":"2024-09-11T04:24:31.209293+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T04:24:33.524789+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:33.524789+0000","last_peered":"2024-09-11T04:24:33.524789+0000","last_clean":"2024-09-11T04:24:33.524789+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:33.524789+0000","last_undegraded":"2024-09-11T04:24:33.524789+0000","last_fullsized":"2024-09-11T04:24:33.524789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31832,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825768,"statfs":{"total":107374182400,"available":107341586432,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31836,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825764,"statfs":{"total":107374182400,"available":107341582336,"internally_reserved":0,"allocated":655360,"data_stored":312318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397528,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31808,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825792,"statfs":{"total":107374182400,"available":107341611008,"internally_reserved":0,"allocated":634880,"data_stored":296933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462937,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825760,"statfs":{"total":107374182400,"available":107341578240,"internally_reserved":0,"allocated":667648,"data_stored":319954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037784,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825756,"statfs":{"total":107374182400,"available":107341574144,"internally_reserved":0,"allocated":671744,"data_stored":319973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":679936,"data_stored":328430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:39.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:39.950 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-11T04:24:38.568520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":306168,"kb_used_data":6708,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048269832,"statfs":{"total":1073741824000,"available":1073428307968,"internally_reserved":0,"allocated":6868992,"data_stored":3321062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001725"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.856625+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:34.856625+0000","last_peered":"2024-09-11T04:24:34.856625+0000","last_clean":"2024-09-11T04:24:34.856625+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:34.856625+0000","last_undegraded":"2024-09-11T04:24:34.856625+0000","last_fullsized":"2024-09-11T04:24:34.856625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.013053+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:35.013053+0000","last_peered":"2024-09-11T04:24:35.013053+0000","last_clean":"2024-09-11T04:24:35.013053+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:35.013053+0000","last_undegraded":"2024-09-11T04:24:35.013053+0000","last_fullsized":"2024-09-11T04:24:35.013053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.090757+0000","last_change":"2024-09-11T04:24:35.090704+0000","last_active":"2024-09-11T04:24:35.090757+0000","last_peered":"2024-09-11T04:24:35.090757+0000","last_clean":"2024-09-11T04:24:35.090757+0000","last_became_active":"2024-09-11T04:24:35.090494+0000","last_became_peered":"2024-09-11T04:24:35.090494+0000","last_unstale":"2024-09-11T04:24:35.090757+0000","last_undegraded":"2024-09-11T04:24:35.090757+0000","last_fullsized":"2024-09-11T04:24:35.090757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:39:33.482975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-11T04:24:34.028224+0000","last_change":"2024-09-11T04:24:33.856626+0000","last_active":"2024-09-11T04:24:33.856626+0000","last_peered":"2024-09-11T04:24:28.164534+0000","last_clean":"2024-09-11T04:24:28.164534+0000","last_became_active":"2024-09-11T04:24:20.790226+0000","last_became_peered":"2024-09-11T04:24:20.790226+0000","last_unstale":"2024-09-11T04:24:34.028224+0000","last_undegraded":"2024-09-11T04:24:34.028224+0000","last_fullsized":"2024-09-11T04:24:34.028224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":98,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859303+0000","last_change":"2024-09-11T04:24:34.859204+0000","last_active":"2024-09-11T04:24:34.859303+0000","last_peered":"2024-09-11T04:24:34.859303+0000","last_clean":"2024-09-11T04:24:34.859303+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:34.859303+0000","last_undegraded":"2024-09-11T04:24:34.859303+0000","last_fullsized":"2024-09-11T04:24:34.859303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:24.877151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":96,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859128+0000","last_change":"2024-09-11T04:24:34.858992+0000","last_active":"2024-09-11T04:24:34.859128+0000","last_peered":"2024-09-11T04:24:34.859128+0000","last_clean":"2024-09-11T04:24:34.859128+0000","last_became_active":"2024-09-11T04:24:34.858543+0000","last_became_peered":"2024-09-11T04:24:34.858543+0000","last_unstale":"2024-09-11T04:24:34.859128+0000","last_undegraded":"2024-09-11T04:24:34.859128+0000","last_fullsized":"2024-09-11T04:24:34.859128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:24:21.576161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.166159+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:35.166159+0000","last_peered":"2024-09-11T04:24:35.166159+0000","last_clean":"2024-09-11T04:24:35.166159+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:35.166159+0000","last_undegraded":"2024-09-11T04:24:35.166159+0000","last_fullsized":"2024-09-11T04:24:35.166159+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T04:24:33.524789+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:33.524789+0000","last_peered":"2024-09-11T04:24:33.524789+0000","last_clean":"2024-09-11T04:24:33.524789+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:33.524789+0000","last_undegraded":"2024-09-11T04:24:33.524789+0000","last_fullsized":"2024-09-11T04:24:33.524789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397529,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31876,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825724,"statfs":{"total":107374182400,"available":107341541376,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462937,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825760,"statfs":{"total":107374182400,"available":107341578240,"internally_reserved":0,"allocated":667648,"data_stored":319954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037784,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825756,"statfs":{"total":107374182400,"available":107341574144,"internally_reserved":0,"allocated":671744,"data_stored":319973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":679936,"data_stored":328430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:39.951 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:40.264 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:40.264 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:40.283 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-11T04:24:38.568520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":306168,"kb_used_data":6708,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048269832,"statfs":{"total":1073741824000,"available":1073428307968,"internally_reserved":0,"allocated":6868992,"data_stored":3321062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001725"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.856625+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:34.856625+0000","last_peered":"2024-09-11T04:24:34.856625+0000","last_clean":"2024-09-11T04:24:34.856625+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:34.856625+0000","last_undegraded":"2024-09-11T04:24:34.856625+0000","last_fullsized":"2024-09-11T04:24:34.856625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.013053+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:35.013053+0000","last_peered":"2024-09-11T04:24:35.013053+0000","last_clean":"2024-09-11T04:24:35.013053+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:35.013053+0000","last_undegraded":"2024-09-11T04:24:35.013053+0000","last_fullsized":"2024-09-11T04:24:35.013053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.090757+0000","last_change":"2024-09-11T04:24:35.090704+0000","last_active":"2024-09-11T04:24:35.090757+0000","last_peered":"2024-09-11T04:24:35.090757+0000","last_clean":"2024-09-11T04:24:35.090757+0000","last_became_active":"2024-09-11T04:24:35.090494+0000","last_became_peered":"2024-09-11T04:24:35.090494+0000","last_unstale":"2024-09-11T04:24:35.090757+0000","last_undegraded":"2024-09-11T04:24:35.090757+0000","last_fullsized":"2024-09-11T04:24:35.090757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:39:33.482975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":60,"state":"peering","last_fresh":"2024-09-11T04:24:34.028224+0000","last_change":"2024-09-11T04:24:33.856626+0000","last_active":"2024-09-11T04:24:33.856626+0000","last_peered":"2024-09-11T04:24:28.164534+0000","last_clean":"2024-09-11T04:24:28.164534+0000","last_became_active":"2024-09-11T04:24:20.790226+0000","last_became_peered":"2024-09-11T04:24:20.790226+0000","last_unstale":"2024-09-11T04:24:34.028224+0000","last_undegraded":"2024-09-11T04:24:34.028224+0000","last_fullsized":"2024-09-11T04:24:34.028224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":98,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859303+0000","last_change":"2024-09-11T04:24:34.859204+0000","last_active":"2024-09-11T04:24:34.859303+0000","last_peered":"2024-09-11T04:24:34.859303+0000","last_clean":"2024-09-11T04:24:34.859303+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:34.859303+0000","last_undegraded":"2024-09-11T04:24:34.859303+0000","last_fullsized":"2024-09-11T04:24:34.859303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:24.877151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":96,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859128+0000","last_change":"2024-09-11T04:24:34.858992+0000","last_active":"2024-09-11T04:24:34.859128+0000","last_peered":"2024-09-11T04:24:34.859128+0000","last_clean":"2024-09-11T04:24:34.859128+0000","last_became_active":"2024-09-11T04:24:34.858543+0000","last_became_peered":"2024-09-11T04:24:34.858543+0000","last_unstale":"2024-09-11T04:24:34.859128+0000","last_undegraded":"2024-09-11T04:24:34.859128+0000","last_fullsized":"2024-09-11T04:24:34.859128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:24:21.576161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.166159+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:35.166159+0000","last_peered":"2024-09-11T04:24:35.166159+0000","last_clean":"2024-09-11T04:24:35.166159+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:35.166159+0000","last_undegraded":"2024-09-11T04:24:35.166159+0000","last_fullsized":"2024-09-11T04:24:35.166159+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":80,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-11T04:24:33.524789+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:33.524789+0000","last_peered":"2024-09-11T04:24:33.524789+0000","last_clean":"2024-09-11T04:24:33.524789+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:33.524789+0000","last_undegraded":"2024-09-11T04:24:33.524789+0000","last_fullsized":"2024-09-11T04:24:33.524789+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397529,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31876,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825724,"statfs":{"total":107374182400,"available":107341541376,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462937,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825760,"statfs":{"total":107374182400,"available":107341578240,"internally_reserved":0,"allocated":667648,"data_stored":319954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037784,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825756,"statfs":{"total":107374182400,"available":107341574144,"internally_reserved":0,"allocated":671744,"data_stored":319973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037785,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":679936,"data_stored":328430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:40.284 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2024-09-11T04:24:40.284 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 70, 'reported_epoch': 60, 'state': 'peering', 'last_fresh': '2024-09-11T04:24:34.028224+0000', 'last_change': '2024-09-11T04:24:33.856626+0000', 'last_active': '2024-09-11T04:24:33.856626+0000', 'last_peered': '2024-09-11T04:24:28.164534+0000', 'last_clean': '2024-09-11T04:24:28.164534+0000', 'last_became_active': '2024-09-11T04:24:20.790226+0000', 'last_became_peered': '2024-09-11T04:24:20.790226+0000', 'last_unstale': '2024-09-11T04:24:34.028224+0000', 'last_undegraded': '2024-09-11T04:24:34.028224+0000', 'last_fullsized': '2024-09-11T04:24:34.028224+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'last_clean_scrub_stamp': '2024-09-11T04:22:53.610870+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-11T04:24:40.284 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-11T04:24:40.647 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:40.648 INFO:teuthology.orchestra.run.smithi002.stdout:{"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":127,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":10,"num_up_osds":10,"osd_up_since":1726028673,"num_in_osds":10,"osd_in_since":1726028556,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"peering","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":313516032,"bytes_avail":1073428307968,"bytes_total":1073741824000,"inactive_pgs_ratio":0.125},"fsmap":{"epoch":1,"btime":"2024-09-11T04:22:33:407386+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-11T04:23:48.556119+0000","services":{}},"progress_events":{}} 2024-09-11T04:24:40.666 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-11T04:24:40.666 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:40.974 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:40.974 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:40.991 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-11T04:24:40.569076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":306228,"kb_used_data":6768,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048269772,"statfs":{"total":1073741824000,"available":1073428246528,"internally_reserved":0,"allocated":6930432,"data_stored":3361018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.718461"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.856625+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:34.856625+0000","last_peered":"2024-09-11T04:24:34.856625+0000","last_clean":"2024-09-11T04:24:34.856625+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:34.856625+0000","last_undegraded":"2024-09-11T04:24:34.856625+0000","last_fullsized":"2024-09-11T04:24:34.856625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.013053+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:35.013053+0000","last_peered":"2024-09-11T04:24:35.013053+0000","last_clean":"2024-09-11T04:24:35.013053+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:35.013053+0000","last_undegraded":"2024-09-11T04:24:35.013053+0000","last_fullsized":"2024-09-11T04:24:35.013053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.090757+0000","last_change":"2024-09-11T04:24:35.090704+0000","last_active":"2024-09-11T04:24:35.090757+0000","last_peered":"2024-09-11T04:24:35.090757+0000","last_clean":"2024-09-11T04:24:35.090757+0000","last_became_active":"2024-09-11T04:24:35.090494+0000","last_became_peered":"2024-09-11T04:24:35.090494+0000","last_unstale":"2024-09-11T04:24:35.090757+0000","last_undegraded":"2024-09-11T04:24:35.090757+0000","last_fullsized":"2024-09-11T04:24:35.090757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:39:33.482975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.015743+0000","last_change":"2024-09-11T04:24:35.015550+0000","last_active":"2024-09-11T04:24:35.015743+0000","last_peered":"2024-09-11T04:24:35.015743+0000","last_clean":"2024-09-11T04:24:35.015743+0000","last_became_active":"2024-09-11T04:24:35.014886+0000","last_became_peered":"2024-09-11T04:24:35.014886+0000","last_unstale":"2024-09-11T04:24:35.015743+0000","last_undegraded":"2024-09-11T04:24:35.015743+0000","last_fullsized":"2024-09-11T04:24:35.015743+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:50:44.930686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":98,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859303+0000","last_change":"2024-09-11T04:24:34.859204+0000","last_active":"2024-09-11T04:24:34.859303+0000","last_peered":"2024-09-11T04:24:34.859303+0000","last_clean":"2024-09-11T04:24:34.859303+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:34.859303+0000","last_undegraded":"2024-09-11T04:24:34.859303+0000","last_fullsized":"2024-09-11T04:24:34.859303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:24.877151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":96,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859128+0000","last_change":"2024-09-11T04:24:34.858992+0000","last_active":"2024-09-11T04:24:34.859128+0000","last_peered":"2024-09-11T04:24:34.859128+0000","last_clean":"2024-09-11T04:24:34.859128+0000","last_became_active":"2024-09-11T04:24:34.858543+0000","last_became_peered":"2024-09-11T04:24:34.858543+0000","last_unstale":"2024-09-11T04:24:34.859128+0000","last_undegraded":"2024-09-11T04:24:34.859128+0000","last_fullsized":"2024-09-11T04:24:34.859128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:24:21.576161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.166159+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:35.166159+0000","last_peered":"2024-09-11T04:24:35.166159+0000","last_clean":"2024-09-11T04:24:35.166159+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:35.166159+0000","last_undegraded":"2024-09-11T04:24:35.166159+0000","last_fullsized":"2024-09-11T04:24:35.166159+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.945850+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:34.945850+0000","last_peered":"2024-09-11T04:24:34.945850+0000","last_clean":"2024-09-11T04:24:34.945850+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:34.945850+0000","last_undegraded":"2024-09-11T04:24:34.945850+0000","last_fullsized":"2024-09-11T04:24:34.945850+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397529,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31876,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825724,"statfs":{"total":107374182400,"available":107341541376,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462937,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037785,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:43.993 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:44.312 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:44.312 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:44.331 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-11T04:24:42.569717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":306228,"kb_used_data":6768,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048269772,"statfs":{"total":1073741824000,"available":1073428246528,"internally_reserved":0,"allocated":6930432,"data_stored":3361018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.716853"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.856625+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:34.856625+0000","last_peered":"2024-09-11T04:24:34.856625+0000","last_clean":"2024-09-11T04:24:34.856625+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:34.856625+0000","last_undegraded":"2024-09-11T04:24:34.856625+0000","last_fullsized":"2024-09-11T04:24:34.856625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.013053+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:35.013053+0000","last_peered":"2024-09-11T04:24:35.013053+0000","last_clean":"2024-09-11T04:24:35.013053+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:35.013053+0000","last_undegraded":"2024-09-11T04:24:35.013053+0000","last_fullsized":"2024-09-11T04:24:35.013053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.090757+0000","last_change":"2024-09-11T04:24:35.090704+0000","last_active":"2024-09-11T04:24:35.090757+0000","last_peered":"2024-09-11T04:24:35.090757+0000","last_clean":"2024-09-11T04:24:35.090757+0000","last_became_active":"2024-09-11T04:24:35.090494+0000","last_became_peered":"2024-09-11T04:24:35.090494+0000","last_unstale":"2024-09-11T04:24:35.090757+0000","last_undegraded":"2024-09-11T04:24:35.090757+0000","last_fullsized":"2024-09-11T04:24:35.090757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:39:33.482975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.015743+0000","last_change":"2024-09-11T04:24:35.015550+0000","last_active":"2024-09-11T04:24:35.015743+0000","last_peered":"2024-09-11T04:24:35.015743+0000","last_clean":"2024-09-11T04:24:35.015743+0000","last_became_active":"2024-09-11T04:24:35.014886+0000","last_became_peered":"2024-09-11T04:24:35.014886+0000","last_unstale":"2024-09-11T04:24:35.015743+0000","last_undegraded":"2024-09-11T04:24:35.015743+0000","last_fullsized":"2024-09-11T04:24:35.015743+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:50:44.930686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":98,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859303+0000","last_change":"2024-09-11T04:24:34.859204+0000","last_active":"2024-09-11T04:24:34.859303+0000","last_peered":"2024-09-11T04:24:34.859303+0000","last_clean":"2024-09-11T04:24:34.859303+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:34.859303+0000","last_undegraded":"2024-09-11T04:24:34.859303+0000","last_fullsized":"2024-09-11T04:24:34.859303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:24.877151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":96,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859128+0000","last_change":"2024-09-11T04:24:34.858992+0000","last_active":"2024-09-11T04:24:34.859128+0000","last_peered":"2024-09-11T04:24:34.859128+0000","last_clean":"2024-09-11T04:24:34.859128+0000","last_became_active":"2024-09-11T04:24:34.858543+0000","last_became_peered":"2024-09-11T04:24:34.858543+0000","last_unstale":"2024-09-11T04:24:34.859128+0000","last_undegraded":"2024-09-11T04:24:34.859128+0000","last_fullsized":"2024-09-11T04:24:34.859128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:24:21.576161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.166159+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:35.166159+0000","last_peered":"2024-09-11T04:24:35.166159+0000","last_clean":"2024-09-11T04:24:35.166159+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:35.166159+0000","last_undegraded":"2024-09-11T04:24:35.166159+0000","last_fullsized":"2024-09-11T04:24:35.166159+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.945850+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:34.945850+0000","last_peered":"2024-09-11T04:24:34.945850+0000","last_clean":"2024-09-11T04:24:34.945850+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:34.945850+0000","last_undegraded":"2024-09-11T04:24:34.945850+0000","last_fullsized":"2024-09-11T04:24:34.945850+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397530,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31876,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825724,"statfs":{"total":107374182400,"available":107341541376,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462938,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037785,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:44.332 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T04:24:44.333 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:44.649 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:44.650 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:44.669 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-11T04:24:44.570236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":306228,"kb_used_data":6768,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048269772,"statfs":{"total":1073741824000,"available":1073428246528,"internally_reserved":0,"allocated":6930432,"data_stored":3361018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002822"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.856625+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:34.856625+0000","last_peered":"2024-09-11T04:24:34.856625+0000","last_clean":"2024-09-11T04:24:34.856625+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:34.856625+0000","last_undegraded":"2024-09-11T04:24:34.856625+0000","last_fullsized":"2024-09-11T04:24:34.856625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.013053+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:35.013053+0000","last_peered":"2024-09-11T04:24:35.013053+0000","last_clean":"2024-09-11T04:24:35.013053+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:35.013053+0000","last_undegraded":"2024-09-11T04:24:35.013053+0000","last_fullsized":"2024-09-11T04:24:35.013053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.090757+0000","last_change":"2024-09-11T04:24:35.090704+0000","last_active":"2024-09-11T04:24:35.090757+0000","last_peered":"2024-09-11T04:24:35.090757+0000","last_clean":"2024-09-11T04:24:35.090757+0000","last_became_active":"2024-09-11T04:24:35.090494+0000","last_became_peered":"2024-09-11T04:24:35.090494+0000","last_unstale":"2024-09-11T04:24:35.090757+0000","last_undegraded":"2024-09-11T04:24:35.090757+0000","last_fullsized":"2024-09-11T04:24:35.090757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:39:33.482975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.015743+0000","last_change":"2024-09-11T04:24:35.015550+0000","last_active":"2024-09-11T04:24:35.015743+0000","last_peered":"2024-09-11T04:24:35.015743+0000","last_clean":"2024-09-11T04:24:35.015743+0000","last_became_active":"2024-09-11T04:24:35.014886+0000","last_became_peered":"2024-09-11T04:24:35.014886+0000","last_unstale":"2024-09-11T04:24:35.015743+0000","last_undegraded":"2024-09-11T04:24:35.015743+0000","last_fullsized":"2024-09-11T04:24:35.015743+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:50:44.930686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":98,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859303+0000","last_change":"2024-09-11T04:24:34.859204+0000","last_active":"2024-09-11T04:24:34.859303+0000","last_peered":"2024-09-11T04:24:34.859303+0000","last_clean":"2024-09-11T04:24:34.859303+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:34.859303+0000","last_undegraded":"2024-09-11T04:24:34.859303+0000","last_fullsized":"2024-09-11T04:24:34.859303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:24.877151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":96,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859128+0000","last_change":"2024-09-11T04:24:34.858992+0000","last_active":"2024-09-11T04:24:34.859128+0000","last_peered":"2024-09-11T04:24:34.859128+0000","last_clean":"2024-09-11T04:24:34.859128+0000","last_became_active":"2024-09-11T04:24:34.858543+0000","last_became_peered":"2024-09-11T04:24:34.858543+0000","last_unstale":"2024-09-11T04:24:34.859128+0000","last_undegraded":"2024-09-11T04:24:34.859128+0000","last_fullsized":"2024-09-11T04:24:34.859128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:24:21.576161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.166159+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:35.166159+0000","last_peered":"2024-09-11T04:24:35.166159+0000","last_clean":"2024-09-11T04:24:35.166159+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:35.166159+0000","last_undegraded":"2024-09-11T04:24:35.166159+0000","last_fullsized":"2024-09-11T04:24:35.166159+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":84,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.945850+0000","last_change":"2024-09-11T04:24:23.104358+0000","last_active":"2024-09-11T04:24:34.945850+0000","last_peered":"2024-09-11T04:24:34.945850+0000","last_clean":"2024-09-11T04:24:34.945850+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:34.945850+0000","last_undegraded":"2024-09-11T04:24:34.945850+0000","last_fullsized":"2024-09-11T04:24:34.945850+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:26.253027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462939,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397530,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31876,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825724,"statfs":{"total":107374182400,"available":107341541376,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462938,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037785,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:44.670 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T04:24:44.975 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:44.975 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":61,"fsid":"6afb2c49-4f7c-48d9-9bd6-caa18d82c311","created":"2024-09-11T04:22:33.407610+0000","modified":"2024-09-11T04:24:34.851930+0000","last_up_change":"2024-09-11T04:24:33.850139+0000","last_in_change":"2024-09-11T04:22:36.496762+0000","flags":"noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800064,"flags_set":["noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-11T04:22:52.970102+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":22,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.80000001192092896,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"86da9381-7ebf-4d49-b779-97f9152b15d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":59,"up_from":60,"up_thru":37,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6801","nonce":3374428017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":3377428017},{"type":"v1","addr":"172.21.15.2:6843","nonce":3377428017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6807","nonce":3374428017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3374428017},{"type":"v1","addr":"172.21.15.2:6805","nonce":3374428017}]},"public_addr":"172.21.15.2:6801/3374428017","cluster_addr":"172.21.15.2:6843/3377428017","heartbeat_back_addr":"172.21.15.2:6807/3374428017","heartbeat_front_addr":"172.21.15.2:6805/3374428017","state":["exists","up"]},{"osd":1,"uuid":"a81b2204-71b5-4a37-b69f-19dbd0999d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":59,"up_from":60,"up_thru":60,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6809","nonce":3277671950}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":3280671950},{"type":"v1","addr":"172.21.15.2:6819","nonce":3280671950}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6815","nonce":3277671950}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3277671950},{"type":"v1","addr":"172.21.15.2:6813","nonce":3277671950}]},"public_addr":"172.21.15.2:6809/3277671950","cluster_addr":"172.21.15.2:6819/3280671950","heartbeat_back_addr":"172.21.15.2:6815/3277671950","heartbeat_front_addr":"172.21.15.2:6813/3277671950","state":["exists","up"]},{"osd":2,"uuid":"df2297b4-4b3c-47a7-b628-11befe5e0a13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":0,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6825","nonce":151425532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6882","nonce":152425532},{"type":"v1","addr":"172.21.15.2:6883","nonce":152425532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6831","nonce":151425532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":151425532},{"type":"v1","addr":"172.21.15.2:6829","nonce":151425532}]},"public_addr":"172.21.15.2:6825/151425532","cluster_addr":"172.21.15.2:6883/152425532","heartbeat_back_addr":"172.21.15.2:6831/151425532","heartbeat_front_addr":"172.21.15.2:6829/151425532","state":["exists","up"]},{"osd":3,"uuid":"91dd038e-a387-4088-93b6-1693e5c5e35e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":60,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6861","nonce":3860158544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6862","nonce":3861158544},{"type":"v1","addr":"172.21.15.2:6864","nonce":3861158544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6871","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6873","nonce":3860158544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":3860158544},{"type":"v1","addr":"172.21.15.2:6869","nonce":3860158544}]},"public_addr":"172.21.15.2:6861/3860158544","cluster_addr":"172.21.15.2:6864/3861158544","heartbeat_back_addr":"172.21.15.2:6873/3860158544","heartbeat_front_addr":"172.21.15.2:6869/3860158544","state":["exists","up"]},{"osd":4,"uuid":"8bcc434f-9122-4417-81aa-3cb6770d7970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":48,"up_from":49,"up_thru":60,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6833","nonce":2837961144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2838961144},{"type":"v1","addr":"172.21.15.2:6811","nonce":2838961144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6839","nonce":2837961144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2837961144},{"type":"v1","addr":"172.21.15.2:6837","nonce":2837961144}]},"public_addr":"172.21.15.2:6833/2837961144","cluster_addr":"172.21.15.2:6811/2838961144","heartbeat_back_addr":"172.21.15.2:6839/2837961144","heartbeat_front_addr":"172.21.15.2:6837/2837961144","state":["exists","up"]},{"osd":5,"uuid":"ceff6b94-d90c-4a0f-8fe0-0ca586ec9c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":48,"up_from":49,"up_thru":49,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6817","nonce":512998079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":513998079},{"type":"v1","addr":"172.21.15.2:6835","nonce":513998079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6823","nonce":512998079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":512998079},{"type":"v1","addr":"172.21.15.2:6821","nonce":512998079}]},"public_addr":"172.21.15.2:6817/512998079","cluster_addr":"172.21.15.2:6835/513998079","heartbeat_back_addr":"172.21.15.2:6823/512998079","heartbeat_front_addr":"172.21.15.2:6821/512998079","state":["exists","up"]},{"osd":6,"uuid":"4089cacf-00bd-46e6-90c0-c99186a7b8bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":46,"up_from":47,"up_thru":56,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6841","nonce":1771532811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6876","nonce":1772532811},{"type":"v1","addr":"172.21.15.2:6877","nonce":1772532811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6847","nonce":1771532811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":1771532811},{"type":"v1","addr":"172.21.15.2:6845","nonce":1771532811}]},"public_addr":"172.21.15.2:6841/1771532811","cluster_addr":"172.21.15.2:6877/1772532811","heartbeat_back_addr":"172.21.15.2:6847/1771532811","heartbeat_front_addr":"172.21.15.2:6845/1771532811","state":["exists","up"]},{"osd":7,"uuid":"3b9b3bcf-ff2f-4d4c-bacd-f52091ffe624","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":46,"up_from":47,"up_thru":47,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6849","nonce":1091593676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1092593676},{"type":"v1","addr":"172.21.15.2:6827","nonce":1092593676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6854","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6855","nonce":1091593676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6852","nonce":1091593676},{"type":"v1","addr":"172.21.15.2:6853","nonce":1091593676}]},"public_addr":"172.21.15.2:6849/1091593676","cluster_addr":"172.21.15.2:6827/1092593676","heartbeat_back_addr":"172.21.15.2:6855/1091593676","heartbeat_front_addr":"172.21.15.2:6853/1091593676","state":["exists","up"]},{"osd":8,"uuid":"dd8535c9-7f50-4b76-8ddb-92d4a777121a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":19,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6875","nonce":2913413062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":2915413062},{"type":"v1","addr":"172.21.15.2:6851","nonce":2915413062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6880","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6881","nonce":2913413062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6878","nonce":2913413062},{"type":"v1","addr":"172.21.15.2:6879","nonce":2913413062}]},"public_addr":"172.21.15.2:6875/2913413062","cluster_addr":"172.21.15.2:6851/2915413062","heartbeat_back_addr":"172.21.15.2:6881/2913413062","heartbeat_front_addr":"172.21.15.2:6879/2913413062","state":["exists","up"]},{"osd":9,"uuid":"3a3c85d9-240a-4ac9-b07e-e9e860d8bcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":39,"up_from":40,"up_thru":60,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6860","nonce":3427009172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3428009172},{"type":"v1","addr":"172.21.15.2:6803","nonce":3428009172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6870","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6872","nonce":3427009172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":3427009172},{"type":"v1","addr":"172.21.15.2:6868","nonce":3427009172}]},"public_addr":"172.21.15.2:6860/3427009172","cluster_addr":"172.21.15.2:6803/3428009172","heartbeat_back_addr":"172.21.15.2:6872/3427009172","heartbeat_front_addr":"172.21.15.2:6868/3427009172","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-09-11T04:24:28.829633+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.054123+0000","dead_epoch":56},{"osd":1,"down_stamp":"2024-09-11T04:24:28.829633+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.288392+0000","dead_epoch":56},{"osd":2,"down_stamp":"2024-09-11T04:24:15.746454+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.952572+0000","dead_epoch":45},{"osd":3,"down_stamp":"2024-09-11T04:24:15.746454+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.123353+0000","dead_epoch":45},{"osd":4,"down_stamp":"2024-09-11T04:24:15.746454+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.494506+0000","dead_epoch":46},{"osd":5,"down_stamp":"2024-09-11T04:24:15.746454+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.314418+0000","dead_epoch":46},{"osd":6,"down_stamp":"2024-09-11T04:24:15.746454+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:42.614299+0000","dead_epoch":43},{"osd":7,"down_stamp":"2024-09-11T04:24:15.746454+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.185779+0000","dead_epoch":45},{"osd":8,"down_stamp":"2024-09-11T04:24:15.746454+0000","laggy_probability":0.50999999046325684,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.013845+0000","dead_epoch":45},{"osd":9,"down_stamp":"2024-09-11T04:24:06.698128+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T04:22:43.252921+0000","dead_epoch":37}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T04:24:45.994 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T04:24:45.994 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:46.158 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:46.158 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:46.158 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:46.174 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T04:24:46.473 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T04:24:46.491 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T04:24:46.491 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:46.642 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:46.642 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:46.642 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:46.658 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T04:24:46.946 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T04:24:46.963 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T04:24:46.964 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:47.110 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:47.110 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:47.110 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:47.125 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T04:24:47.415 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T04:24:47.434 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T04:24:47.434 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:47.589 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:47.589 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:47.589 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:47.606 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T04:24:47.905 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T04:24:47.923 INFO:tasks.ceph:Scrubbing osd.4 2024-09-11T04:24:47.924 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:48.073 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:48.073 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:48.073 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:48.089 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-11T04:24:48.385 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 4 to deep-scrub 2024-09-11T04:24:48.404 INFO:tasks.ceph:Scrubbing osd.5 2024-09-11T04:24:48.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:48.562 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:48.562 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:48.562 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:48.579 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-11T04:24:48.882 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 5 to deep-scrub 2024-09-11T04:24:48.900 INFO:tasks.ceph:Scrubbing osd.6 2024-09-11T04:24:48.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:49.043 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:49.043 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:49.043 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:49.058 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-11T04:24:49.360 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 6 to deep-scrub 2024-09-11T04:24:49.380 INFO:tasks.ceph:Scrubbing osd.7 2024-09-11T04:24:49.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:49.534 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:49.534 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:49.534 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:49.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-11T04:24:49.856 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 7 to deep-scrub 2024-09-11T04:24:49.876 INFO:tasks.ceph:Scrubbing osd.8 2024-09-11T04:24:49.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:50.031 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:50.031 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:50.031 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:50.048 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-11T04:24:50.347 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 8 to deep-scrub 2024-09-11T04:24:50.365 INFO:tasks.ceph:Scrubbing osd.9 2024-09-11T04:24:50.365 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-11T04:24:50.518 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-11T04:24:50.518 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T04:24:50.518 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-11T04:24:50.535 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-11T04:24:50.836 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 9 to deep-scrub 2024-09-11T04:24:50.857 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:24:51.159 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:24:51.160 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:24:51.178 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-11T04:24:50.571295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":306224,"kb_used_data":6768,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048269776,"statfs":{"total":1073741824000,"available":1073428250624,"internally_reserved":0,"allocated":6930432,"data_stored":3361018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002775"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.856625+0000","last_change":"2024-09-11T04:24:22.795213+0000","last_active":"2024-09-11T04:24:34.856625+0000","last_peered":"2024-09-11T04:24:34.856625+0000","last_clean":"2024-09-11T04:24:34.856625+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:34.856625+0000","last_undegraded":"2024-09-11T04:24:34.856625+0000","last_fullsized":"2024-09-11T04:24:34.856625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:18.926697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.013053+0000","last_change":"2024-09-11T04:24:22.798931+0000","last_active":"2024-09-11T04:24:35.013053+0000","last_peered":"2024-09-11T04:24:35.013053+0000","last_clean":"2024-09-11T04:24:35.013053+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:35.013053+0000","last_undegraded":"2024-09-11T04:24:35.013053+0000","last_fullsized":"2024-09-11T04:24:35.013053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:59.211198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.090757+0000","last_change":"2024-09-11T04:24:35.090704+0000","last_active":"2024-09-11T04:24:35.090757+0000","last_peered":"2024-09-11T04:24:35.090757+0000","last_clean":"2024-09-11T04:24:35.090757+0000","last_became_active":"2024-09-11T04:24:35.090494+0000","last_became_peered":"2024-09-11T04:24:35.090494+0000","last_unstale":"2024-09-11T04:24:35.090757+0000","last_undegraded":"2024-09-11T04:24:35.090757+0000","last_fullsized":"2024-09-11T04:24:35.090757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:39:33.482975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":87,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:47.904248+0000","last_change":"2024-09-11T04:24:47.904248+0000","last_active":"2024-09-11T04:24:47.904248+0000","last_peered":"2024-09-11T04:24:47.904248+0000","last_clean":"2024-09-11T04:24:47.904248+0000","last_became_active":"2024-09-11T04:24:35.014886+0000","last_became_peered":"2024-09-11T04:24:35.014886+0000","last_unstale":"2024-09-11T04:24:47.904248+0000","last_undegraded":"2024-09-11T04:24:47.904248+0000","last_fullsized":"2024-09-11T04:24:47.904248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:47.904085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:47.904085+0000","last_clean_scrub_stamp":"2024-09-11T04:24:47.904085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:42.869816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'1","reported_seq":98,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859303+0000","last_change":"2024-09-11T04:24:34.859204+0000","last_active":"2024-09-11T04:24:34.859303+0000","last_peered":"2024-09-11T04:24:34.859303+0000","last_clean":"2024-09-11T04:24:34.859303+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:34.859303+0000","last_undegraded":"2024-09-11T04:24:34.859303+0000","last_fullsized":"2024-09-11T04:24:34.859303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:24.877151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'2","reported_seq":96,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:34.859128+0000","last_change":"2024-09-11T04:24:34.858992+0000","last_active":"2024-09-11T04:24:34.859128+0000","last_peered":"2024-09-11T04:24:34.859128+0000","last_clean":"2024-09-11T04:24:34.859128+0000","last_became_active":"2024-09-11T04:24:34.858543+0000","last_became_peered":"2024-09-11T04:24:34.858543+0000","last_unstale":"2024-09-11T04:24:34.859128+0000","last_undegraded":"2024-09-11T04:24:34.859128+0000","last_fullsized":"2024-09-11T04:24:34.859128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:24:21.576161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:35.166159+0000","last_change":"2024-09-11T04:24:20.790625+0000","last_active":"2024-09-11T04:24:35.166159+0000","last_peered":"2024-09-11T04:24:35.166159+0000","last_clean":"2024-09-11T04:24:35.166159+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:35.166159+0000","last_undegraded":"2024-09-11T04:24:35.166159+0000","last_fullsized":"2024-09-11T04:24:35.166159+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:22:53.610870+0000","last_clean_scrub_stamp":"2024-09-11T04:22:53.610870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:44:29.486657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":92,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:49.281730+0000","last_change":"2024-09-11T04:24:49.281730+0000","last_active":"2024-09-11T04:24:49.281730+0000","last_peered":"2024-09-11T04:24:49.281730+0000","last_clean":"2024-09-11T04:24:49.281730+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:49.281730+0000","last_undegraded":"2024-09-11T04:24:49.281730+0000","last_fullsized":"2024-09-11T04:24:49.281730+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:49.281607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:49.281607+0000","last_clean_scrub_stamp":"2024-09-11T04:24:49.281607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:37.656662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462938,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31872,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825728,"statfs":{"total":107374182400,"available":107341545472,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397531,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31876,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825724,"statfs":{"total":107374182400,"available":107341541376,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462939,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037787,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:24:51.182 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-11T04:22:53.610870+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=22, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T04:24:51.182 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-11T04:22:53.610870+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=22, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T04:24:51.182 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-11T04:22:53.610870+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=22, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T04:24:51.182 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-11T04:22:53.610870+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=22, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T04:24:51.182 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-11T04:22:53.610870+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=22, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T04:24:51.182 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-11T04:22:53.610870+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=22, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=4, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T04:24:51.183 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T04:25:11.184 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T04:25:11.504 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-11T04:25:11.505 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-11T04:25:11.522 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-11T04:25:10.576653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":306156,"kb_used_data":6768,"kb_used_omap":15,"kb_used_meta":299184,"kb_avail":1048269844,"statfs":{"total":1073741824000,"available":1073428320256,"internally_reserved":0,"allocated":6930432,"data_stored":3361018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15944,"internal_metadata":306364856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:49.127617+0000","last_change":"2024-09-11T04:24:49.127617+0000","last_active":"2024-09-11T04:24:49.127617+0000","last_peered":"2024-09-11T04:24:49.127617+0000","last_clean":"2024-09-11T04:24:49.127617+0000","last_became_active":"2024-09-11T04:24:22.794449+0000","last_became_peered":"2024-09-11T04:24:22.794449+0000","last_unstale":"2024-09-11T04:24:49.127617+0000","last_undegraded":"2024-09-11T04:24:49.127617+0000","last_fullsized":"2024-09-11T04:24:49.127617+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:49.127553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:49.127553+0000","last_clean_scrub_stamp":"2024-09-11T04:24:49.127553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:25.088033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00057948400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":111,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:49.992646+0000","last_change":"2024-09-11T04:24:49.992561+0000","last_active":"2024-09-11T04:24:49.992646+0000","last_peered":"2024-09-11T04:24:49.992646+0000","last_clean":"2024-09-11T04:24:49.992646+0000","last_became_active":"2024-09-11T04:24:22.798067+0000","last_became_peered":"2024-09-11T04:24:22.798067+0000","last_unstale":"2024-09-11T04:24:49.992646+0000","last_undegraded":"2024-09-11T04:24:49.992646+0000","last_fullsized":"2024-09-11T04:24:49.992646+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:49.992483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:49.992483+0000","last_clean_scrub_stamp":"2024-09-11T04:24:49.992483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:46:31.091380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055793699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":113,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:51.116438+0000","last_change":"2024-09-11T04:24:51.116438+0000","last_active":"2024-09-11T04:24:51.116438+0000","last_peered":"2024-09-11T04:24:51.116438+0000","last_clean":"2024-09-11T04:24:51.116438+0000","last_became_active":"2024-09-11T04:24:35.090494+0000","last_became_peered":"2024-09-11T04:24:35.090494+0000","last_unstale":"2024-09-11T04:24:51.116438+0000","last_undegraded":"2024-09-11T04:24:51.116438+0000","last_fullsized":"2024-09-11T04:24:51.116438+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:51.116362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:51.116362+0000","last_clean_scrub_stamp":"2024-09-11T04:24:51.116362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:45:14.771513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046493000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":87,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:47.904248+0000","last_change":"2024-09-11T04:24:47.904248+0000","last_active":"2024-09-11T04:24:47.904248+0000","last_peered":"2024-09-11T04:24:47.904248+0000","last_clean":"2024-09-11T04:24:47.904248+0000","last_became_active":"2024-09-11T04:24:35.014886+0000","last_became_peered":"2024-09-11T04:24:35.014886+0000","last_unstale":"2024-09-11T04:24:47.904248+0000","last_undegraded":"2024-09-11T04:24:47.904248+0000","last_fullsized":"2024-09-11T04:24:47.904248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:47.904085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:47.904085+0000","last_clean_scrub_stamp":"2024-09-11T04:24:47.904085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:42.869816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"61'2","reported_seq":107,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:48.121044+0000","last_change":"2024-09-11T04:24:48.121044+0000","last_active":"2024-09-11T04:24:48.121044+0000","last_peered":"2024-09-11T04:24:48.121044+0000","last_clean":"2024-09-11T04:24:48.121044+0000","last_became_active":"2024-09-11T04:24:34.858741+0000","last_became_peered":"2024-09-11T04:24:34.858741+0000","last_unstale":"2024-09-11T04:24:48.121044+0000","last_undegraded":"2024-09-11T04:24:48.121044+0000","last_fullsized":"2024-09-11T04:24:48.121044+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-11T04:24:48.120997+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-11T04:24:48.120997+0000","last_clean_scrub_stamp":"2024-09-11T04:24:48.120997+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:47:05.139064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"61'3","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:48.709909+0000","last_change":"2024-09-11T04:24:48.709909+0000","last_active":"2024-09-11T04:24:48.709909+0000","last_peered":"2024-09-11T04:24:48.709909+0000","last_clean":"2024-09-11T04:24:48.709909+0000","last_became_active":"2024-09-11T04:24:34.858543+0000","last_became_peered":"2024-09-11T04:24:34.858543+0000","last_unstale":"2024-09-11T04:24:48.709909+0000","last_undegraded":"2024-09-11T04:24:48.709909+0000","last_fullsized":"2024-09-11T04:24:48.709909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-11T04:24:48.709828+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-11T04:24:48.709828+0000","last_clean_scrub_stamp":"2024-09-11T04:24:48.709828+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:13.572297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001965961,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:50.330587+0000","last_change":"2024-09-11T04:24:50.330587+0000","last_active":"2024-09-11T04:24:50.330587+0000","last_peered":"2024-09-11T04:24:50.330587+0000","last_clean":"2024-09-11T04:24:50.330587+0000","last_became_active":"2024-09-11T04:24:20.789853+0000","last_became_peered":"2024-09-11T04:24:20.789853+0000","last_unstale":"2024-09-11T04:24:50.330587+0000","last_undegraded":"2024-09-11T04:24:50.330587+0000","last_fullsized":"2024-09-11T04:24:50.330587+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:50.330463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:50.330463+0000","last_clean_scrub_stamp":"2024-09-11T04:24:50.330463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:55.930791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053399999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":92,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T04:24:49.281730+0000","last_change":"2024-09-11T04:24:49.281730+0000","last_active":"2024-09-11T04:24:49.281730+0000","last_peered":"2024-09-11T04:24:49.281730+0000","last_clean":"2024-09-11T04:24:49.281730+0000","last_became_active":"2024-09-11T04:24:23.103519+0000","last_became_peered":"2024-09-11T04:24:23.103519+0000","last_unstale":"2024-09-11T04:24:49.281730+0000","last_undegraded":"2024-09-11T04:24:49.281730+0000","last_fullsized":"2024-09-11T04:24:49.281730+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T04:24:49.281607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T04:24:49.281607+0000","last_clean_scrub_stamp":"2024-09-11T04:24:49.281607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:37.656662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462942,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":49,"seq":210453397535,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462943,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27712,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829888,"statfs":{"total":107374182400,"available":107345805312,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037791,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":696320,"data_stored":336117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037792,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":692224,"data_stored":336098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T04:25:11.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T04:25:11.842 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T04:25:11.842 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T04:25:11.843 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T04:25:11.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:11.925 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T04:25:11.926 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T04:25:11.926 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:12.010 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T04:25:12.011 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T04:25:12.011 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:12.098 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T04:25:12.099 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T04:25:12.099 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:12.184 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T04:25:12.184 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T04:25:12.185 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:12.264 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T04:25:12.264 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-11T04:25:12.264 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:12.331 INFO:tasks.ceph.osd.5:Stopped 2024-09-11T04:25:12.332 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-11T04:25:12.332 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:12.402 INFO:tasks.ceph.osd.6:Stopped 2024-09-11T04:25:12.403 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-11T04:25:12.403 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:12.474 INFO:tasks.ceph.osd.7:Stopped 2024-09-11T04:25:12.474 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-11T04:25:12.474 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:13.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2024-09-11T04:25:18.576 INFO:tasks.ceph.osd.8:Stopped 2024-09-11T04:25:18.576 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-11T04:25:18.576 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:18.631 INFO:tasks.ceph.osd.9:Stopped 2024-09-11T04:25:18.631 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T04:25:18.632 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T04:25:18.632 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:18.664 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T04:25:18.665 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T04:25:18.665 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T04:25:18.665 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T04:25:18.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~5s 2024-09-11T04:25:18.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2024-09-11T04:25:18.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-11T04:25:18.718 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T04:25:18.718 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T04:25:18.718 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-11T04:25:18.776 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:25:18.776 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T04:25:19.140 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:25:19.140 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T04:25:19.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:25:19.329 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T04:25:19.496 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:25:19.497 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T04:25:19.647 INFO:tasks.ceph:Archiving mon data... 2024-09-11T04:25:19.648 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900128/data/mon.a.tgz 2024-09-11T04:25:19.648 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-09-11T04:25:19.654 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.rl1GxplF4k 2024-09-11T04:25:19.654 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rl1GxplF4k 2024-09-11T04:25:19.767 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.rl1GxplF4k 2024-09-11T04:25:19.834 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.rl1GxplF4k is 222KB 2024-09-11T04:25:19.896 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.rl1GxplF4k 2024-09-11T04:25:19.903 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T04:25:19.903 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T04:25:20.006 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T04:25:20.006 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T04:25:20.008 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900128/remote/smithi002/crash 2024-09-11T04:25:20.008 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T04:25:20.062 INFO:tasks.ceph:Compressing logs... 2024-09-11T04:25:20.062 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T04:25:20.126 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36392.log 2024-09-11T04:25:20.127 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29547.log 2024-09-11T04:25:20.127 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36392.log.gz 2024-09-11T04:25:20.128 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36699.log 2024-09-11T04:25:20.130 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36453.log 2024-09-11T04:25:20.130 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29547.log.gz 2024-09-11T04:25:20.130 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.36699.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39055.log 2024-09-11T04:25:20.130 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36699.log.gz 2024-09-11T04:25:20.130 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36366.log 2024-09-11T04:25:20.130 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36453.log: /var/log/ceph/ceph-client.admin.39055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36453.log.gz 2024-09-11T04:25:20.130 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39055.log.gz 2024-09-11T04:25:20.131 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38431.log 2024-09-11T04:25:20.131 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36294.log 2024-09-11T04:25:20.131 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36366.log.gz 2024-09-11T04:25:20.131 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38431.log.gz 2024-09-11T04:25:20.131 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29523.log 2024-09-11T04:25:20.136 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36294.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T04:25:20.136 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36294.log.gz 2024-09-11T04:25:20.136 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29523.log.gz 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36728.log 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36689.log 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37303.log 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36728.log.gz 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36220.log 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36689.log.gz 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36450.log 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37303.log.gz 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37649.log 2024-09-11T04:25:20.137 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36220.log.gz 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38116.log 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36450.log.gz 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37649.log.gz 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37748.log 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38821.log 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38116.log.gz 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37748.log.gz 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37524.log 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38821.log.gz 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37524.log.gz 2024-09-11T04:25:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-11T04:25:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T04:25:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37452.log 2024-09-11T04:25:20.142 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37921.log 2024-09-11T04:25:20.142 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37452.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.37452.log.gz /var/log/ceph/ceph-client.admin.38044.log 2024-09-11T04:25:20.142 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-11T04:25:20.142 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37921.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37921.log.gz 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36668.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36340.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36442.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36668.log.gz 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38561.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36340.log.gz 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36442.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36442.log.gz 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38405.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38561.log.gz 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36446.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.38405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38951.log 2024-09-11T04:25:20.143 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38405.log.gz 2024-09-11T04:25:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36446.log.gz 2024-09-11T04:25:20.145 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37427.log 2024-09-11T04:25:20.146 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36938.log 2024-09-11T04:25:20.146 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38951.log.gz 2024-09-11T04:25:20.146 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37427.log.gz 2024-09-11T04:25:20.147 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39160.log 2024-09-11T04:25:20.148 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37403.log 2024-09-11T04:25:20.148 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36938.log: /var/log/ceph/ceph-client.admin.39160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39160.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36938.log.gz 2024-09-11T04:25:20.148 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-11T04:25:20.148 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36688.log 2024-09-11T04:25:20.150 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37797.log 2024-09-11T04:25:20.151 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37403.log.gz 2024-09-11T04:25:20.151 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39081.log 2024-09-11T04:25:20.151 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36688.log.gz 2024-09-11T04:25:20.151 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37797.log.gz 2024-09-11T04:25:20.151 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36454.log 2024-09-11T04:25:20.151 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38353.log 2024-09-11T04:25:20.151 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39081.log.gz 2024-09-11T04:25:20.153 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36454.log.gz 2024-09-11T04:25:20.154 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38613.log 2024-09-11T04:25:20.154 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36990.log 2024-09-11T04:25:20.154 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38353.log.gz 2024-09-11T04:25:20.154 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38613.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38166.log 2024-09-11T04:25:20.154 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38613.log.gz 2024-09-11T04:25:20.154 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36990.log.gzgzip 2024-09-11T04:25:20.154 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37846.log 2024-09-11T04:25:20.157 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37476.log 2024-09-11T04:25:20.157 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38166.log: /var/log/ceph/ceph-client.admin.37846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38166.log.gz 2024-09-11T04:25:20.157 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37846.log.gz 2024-09-11T04:25:20.157 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29674.log 2024-09-11T04:25:20.157 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37674.log 2024-09-11T04:25:20.157 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37476.log: /var/log/ceph/ceph-client.admin.29674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37476.log.gz 2024-09-11T04:25:20.158 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29674.log.gz 2024-09-11T04:25:20.158 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38379.log 2024-09-11T04:25:20.158 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38509.log 2024-09-11T04:25:20.158 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37674.log.gz 2024-09-11T04:25:20.158 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38379.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38795.log 2024-09-11T04:25:20.158 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38379.log.gz 2024-09-11T04:25:20.159 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38509.log.gz 2024-09-11T04:25:20.160 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38327.log 2024-09-11T04:25:20.160 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T04:25:20.160 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38795.log.gz 2024-09-11T04:25:20.160 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38327.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37131.log 2024-09-11T04:25:20.160 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38327.log.gz 2024-09-11T04:25:20.169 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37996.log 2024-09-11T04:25:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37131.log.gz 2024-09-11T04:25:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29316.log 2024-09-11T04:25:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37996.log.gz 2024-09-11T04:25:20.171 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-11T04:25:20.172 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29316.log.gz 2024-09-11T04:25:20.173 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38457.log 2024-09-11T04:25:20.189 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37328.log 2024-09-11T04:25:20.190 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38457.log.gz 2024-09-11T04:25:20.193 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29499.log 2024-09-11T04:25:20.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37328.log.gz 2024-09-11T04:25:20.194 INFO:teuthology.orchestra.run.smithi002.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T04:25:20.194 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37574.log 2024-09-11T04:25:20.195 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29499.log.gz 2024-09-11T04:25:20.196 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36436.log 2024-09-11T04:25:20.196 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37574.log.gz 2024-09-11T04:25:20.197 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37871.log 2024-09-11T04:25:20.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36436.log.gz 2024-09-11T04:25:20.198 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29381.log 2024-09-11T04:25:20.199 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37871.log.gz 2024-09-11T04:25:20.199 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38587.log 2024-09-11T04:25:20.200 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29381.log.gz 2024-09-11T04:25:20.200 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T04:25:20.201 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38587.log.gz 2024-09-11T04:25:20.201 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37599.log 2024-09-11T04:25:20.205 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.1.log: 92.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-11T04:25:20.211 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-11T04:25:20.211 INFO:teuthology.orchestra.run.smithi002.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T04:25:20.212 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36314.log 2024-09-11T04:25:20.212 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38240.log 2024-09-11T04:25:20.213 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37599.log.gz 2024-09-11T04:25:20.213 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38068.log 2024-09-11T04:25:20.213 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36314.log.gz 2024-09-11T04:25:20.226 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36702.log 2024-09-11T04:25:20.226 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38240.log.gz/var/log/ceph/ceph-client.admin.38068.log: 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38068.log.gz 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38216.log 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36702.log.gz 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37231.log 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38216.log.gz 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37821.log 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37231.log.gz 2024-09-11T04:25:20.227 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38639.log 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29643.log 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37821.log: /var/log/ceph/ceph-client.admin.38639.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38639.log.gz 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37821.log.gz 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29643.log.gz 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36716.log 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38092.log 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37945.log 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36716.log.gz 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38092.log.gz 2024-09-11T04:25:20.228 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37945.log.gz 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21387.log 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29619.log 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.tmp-client.admin.21387.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21387.log.gz 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29475.log 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39107.log 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29619.log: /var/log/ceph/ceph-client.admin.29475.log: 0.0% 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37624.log 2024-09-11T04:25:20.229 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29475.log.gz 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29619.log.gz 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39107.log.gz 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37279.log 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38301.log 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37624.log: /var/log/ceph/ceph-client.admin.37279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37624.log.gz 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36691.log 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37279.log.gz 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36244.log 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38301.log: /var/log/ceph/ceph-client.admin.36691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38301.log.gz 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38020.log 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36691.log.gz 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37155.log 2024-09-11T04:25:20.230 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36244.log.gz 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37500.log 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38020.log: /var/log/ceph/ceph-client.admin.37155.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29729.log 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37155.log.gz 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38020.log.gz 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37500.log.gz 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36964.log 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29729.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29729.log.gz 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36964.log.gz 2024-09-11T04:25:20.231 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37207.log 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38717.log 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.37207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36443.log 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37207.log.gz 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38717.log.gz 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38535.log 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36269.log 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36443.log.gz 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38535.log.gz 2024-09-11T04:25:20.238 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38275.log 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36269.log: /var/log/ceph/ceph-client.admin.38275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36269.log.gz 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38191.log 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38275.log.gz 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.38191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37698.log 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38191.log.gz 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29451.log 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38769.log 2024-09-11T04:25:20.239 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37698.log.gz 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29451.log.gz 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38743.log 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38769.log.gz 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29929.log 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38743.log.gz 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37255.log 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38483.log 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29929.log.gz 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37255.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37103.log 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-11T04:25:20.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38483.log.gz 2024-09-11T04:25:20.241 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38847.log 2024-09-11T04:25:20.244 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36439.log 2024-09-11T04:25:20.244 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37103.log: /var/log/ceph/ceph-client.admin.38847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37103.log.gz 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38847.log.gz 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36706.log 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36445.log 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36439.log: /var/log/ceph/ceph-client.admin.36706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36439.log.gz 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36706.log.gz 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38141.log 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36445.log: /var/log/ceph/ceph-client.admin.38873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36445.log.gz 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29571.log 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-11T04:25:20.245 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38141.log: /var/log/ceph/ceph-client.admin.29571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38141.log.gz 2024-09-11T04:25:20.246 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29571.log.gz 2024-09-11T04:25:20.246 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37971.log 2024-09-11T04:25:20.246 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37378.log 2024-09-11T04:25:20.246 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.37971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37971.log.gz 2024-09-11T04:25:20.248 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39029.log 2024-09-11T04:25:20.248 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37378.log.gz 2024-09-11T04:25:20.248 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37896.log 2024-09-11T04:25:20.248 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T04:25:20.248 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39029.log.gz 2024-09-11T04:25:20.249 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-09-11T04:25:20.249 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38691.log 2024-09-11T04:25:20.249 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37896.log: /var/log/ceph/ceph-client.admin.38899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37896.log.gz 2024-09-11T04:25:20.249 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-09-11T04:25:20.249 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37773.log 2024-09-11T04:25:20.251 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37353.log 2024-09-11T04:25:20.251 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38691.log: /var/log/ceph/ceph-client.admin.37773.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38691.log.gz 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37773.log.gz 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36670.log 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37723.log 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37353.log: /var/log/ceph/ceph-client.admin.36670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37549.log 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37353.log.gz 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0%/var/log/ceph/ceph-client.admin.37723.log: -- replaced with /var/log/ceph/ceph-client.admin.36670.log.gz 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr:gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37723.log.gz --verbose 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr: -- /var/log/ceph/ceph-client.admin.29595.log 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37549.log.gz 2024-09-11T04:25:20.252 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.29595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29595.log.gz 2024-09-11T04:25:20.254 INFO:teuthology.orchestra.run.smithi002.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T04:25:20.279 INFO:teuthology.orchestra.run.smithi002.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T04:25:20.291 INFO:teuthology.orchestra.run.smithi002.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T04:25:20.294 INFO:teuthology.orchestra.run.smithi002.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-11T04:25:20.305 INFO:teuthology.orchestra.run.smithi002.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-11T04:25:20.306 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-11T04:25:20.307 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m0.199s 2024-09-11T04:25:20.307 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m0.975s 2024-09-11T04:25:20.307 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.098s 2024-09-11T04:25:20.307 INFO:tasks.ceph:Archiving logs... 2024-09-11T04:25:20.307 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900128/remote/smithi002/log 2024-09-11T04:25:20.308 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T04:25:20.557 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T04:25:20.573 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T04:25:20.574 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T04:25:20.610 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T04:25:20.618 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T04:25:20.619 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T04:25:20.645 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T04:25:20.645 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2024-09-11T04:25:20.645 INFO:teuthology.orchestra.run.smithi002.stdout:+hv01.front.sepi 67.205.162.81 3 u 32 64 37 0.117 -1.004 0.953 2024-09-11T04:25:20.645 INFO:teuthology.orchestra.run.smithi002.stdout:+hv02.front.sepi 63.231.80.2 3 u 38 64 37 0.148 +1.114 0.937 2024-09-11T04:25:20.645 INFO:teuthology.orchestra.run.smithi002.stdout:*hv03.front.sepi 74.6.168.72 3 u 42 64 37 0.097 -1.290 1.043 2024-09-11T04:25:20.645 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T04:25:20.646 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T04:25:20.655 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T04:25:20.656 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T04:25:20.673 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T04:25:20.707 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T04:25:20.741 INFO:teuthology.task.internal:Duration was 765.552414 seconds 2024-09-11T04:25:20.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T04:25:20.774 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T04:25:20.774 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T04:25:20.814 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T04:25:20.815 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-09-11T04:25:20.815 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T04:25:20.866 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T04:25:20.866 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T04:25:20.936 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T04:25:20.936 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T04:25:21.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T04:25:21.046 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T04:25:21.046 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T04:25:21.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T04:25:21.075 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T04:25:21.112 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-09-11T04:25:21.126 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T04:25:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:25:21.173 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T04:25:21.188 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T04:25:21.190 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900128/remote/smithi002 2024-09-11T04:25:21.190 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T04:25:21.239 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T04:25:21.239 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T04:25:21.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T04:25:21.294 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T04:25:21.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T04:25:21.316 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T04:25:21.316 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T04:25:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 04:25 /home/ubuntu/cephtest 2024-09-11T04:25:21.331 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T04:25:21.350 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T04:25:21.418 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} duration: 765.5524137020111 flavor: default owner: scheduled_hyelloji@teuthology success: true 2024-09-11T04:25:21.418 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T04:25:21.506 INFO:teuthology.run:pass