2024-04-23T14:21:46.918 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T14:21:46.918 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} --owner scheduled_yuriw@teuthology --name yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603 -- /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603/orig.config.yaml 2024-04-23T14:21:46.944 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T14:21:47.043 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603 branch: wip-yuri3-testing-2024-04-05-0825 description: rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false job_id: '7669603' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri3-testing-2024-04-05-0825 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb workunit: branch: wip-yuri3-testing-2024-04-05-0825 sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 seed: 3753 sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2024-04-05-0825 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIAH+D4TracU+auAUik3ADNJF4fsvqun5e8LmdeYNNZ8K8LVqzYlnMA7F/LeegIz1oECQ2fYnnachO48W8CKcI= tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: all: - rados/test_health_warnings.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_14:14:08 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T14:21:47.043 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa; will attempt to use it 2024-04-23T14:21:47.044 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks 2024-04-23T14:21:47.044 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T14:21:47.046 INFO:teuthology.task.internal:Checking packages... 2024-04-23T14:21:47.069 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5d349943c59c9485df060d6adb0594f3940ec0eb' 2024-04-23T14:21:47.069 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T14:21:47.069 INFO:teuthology.packaging:ref: None 2024-04-23T14:21:47.069 INFO:teuthology.packaging:tag: None 2024-04-23T14:21:47.069 INFO:teuthology.packaging:branch: wip-yuri3-testing-2024-04-05-0825 2024-04-23T14:21:47.069 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T14:21:47.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri3-testing-2024-04-05-0825 2024-04-23T14:21:47.221 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3211-g5d349943-1jammy 2024-04-23T14:21:47.223 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T14:21:47.229 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T14:21:47.229 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T14:21:47.242 INFO:teuthology.task.internal:Saving configuration 2024-04-23T14:21:47.253 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T14:21:47.260 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T14:21:47.287 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:16:18.533758', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIu7t0rtmz4q1dybCdK2GnungBrePrsRSTEaCHloVd0a'} 2024-04-23T14:21:47.287 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T14:21:47.294 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2024-04-23T14:21:47.294 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T14:21:47.335 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8411d38f70>, signals=[15]) 2024-04-23T14:21:47.336 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T14:21:47.342 INFO:teuthology.task.internal:Opening connections... 2024-04-23T14:21:47.342 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:21:47.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:21:47.426 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T14:21:47.434 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-04-23T14:21:47.527 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-04-23T14:21:47.527 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-04-23T14:21:47.575 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Ubuntu" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="22.04" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_CODENAME=jammy 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:ID=ubuntu 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE=debian 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-23T14:21:47.576 INFO:teuthology.orchestra.run.smithi084.stdout:UBUNTU_CODENAME=jammy 2024-04-23T14:21:47.577 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-04-23T14:21:47.597 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T14:21:47.608 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T14:21:47.616 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T14:21:47.616 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-04-23T14:21:47.623 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T14:21:47.630 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T14:21:47.631 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-04-23T14:21:47.673 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T14:21:47.703 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T14:21:47.719 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-23T14:21:47.720 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-23T14:21:47.720 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T14:21:47.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:21:47.725 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-04-23T14:21:47.772 INFO:teuthology.orchestra.run.smithi084.stdout:5.15.0-56-generic 2024-04-23T14:21:47.772 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.15.0-56-generic 2024-04-23T14:21:47.772 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get clean 2024-04-23T14:21:47.911 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2024-04-23T14:21:48.054 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T14:21:48.055 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-23T14:21:48.056 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-23T14:21:48.146 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-23T14:21:48.303 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-23T14:21:48.371 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-23T14:21:48.392 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,388 kB] 2024-04-23T14:21:48.445 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-23T14:21:48.447 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,608 kB] 2024-04-23T14:21:48.449 INFO:teuthology.orchestra.run.smithi084.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-23T14:21:48.449 INFO:teuthology.orchestra.run.smithi084.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-23T14:21:48.450 INFO:teuthology.orchestra.run.smithi084.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,762 kB] 2024-04-23T14:21:48.481 INFO:teuthology.orchestra.run.smithi084.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-23T14:21:48.485 INFO:teuthology.orchestra.run.smithi084.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-23T14:21:48.485 INFO:teuthology.orchestra.run.smithi084.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-23T14:21:48.497 INFO:teuthology.orchestra.run.smithi084.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-23T14:21:48.528 INFO:teuthology.orchestra.run.smithi084.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-23T14:21:48.528 INFO:teuthology.orchestra.run.smithi084.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-23T14:21:48.529 INFO:teuthology.orchestra.run.smithi084.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-23T14:21:48.531 INFO:teuthology.orchestra.run.smithi084.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-23T14:21:48.531 INFO:teuthology.orchestra.run.smithi084.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-23T14:21:48.533 INFO:teuthology.orchestra.run.smithi084.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-23T14:21:48.533 INFO:teuthology.orchestra.run.smithi084.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-23T14:21:48.534 INFO:teuthology.orchestra.run.smithi084.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-23T14:21:48.534 INFO:teuthology.orchestra.run.smithi084.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,826 kB] 2024-04-23T14:21:48.542 INFO:teuthology.orchestra.run.smithi084.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-23T14:21:48.555 INFO:teuthology.orchestra.run.smithi084.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [310 kB] 2024-04-23T14:21:48.558 INFO:teuthology.orchestra.run.smithi084.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-23T14:21:48.565 INFO:teuthology.orchestra.run.smithi084.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-23T14:21:48.571 INFO:teuthology.orchestra.run.smithi084.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-23T14:21:48.579 INFO:teuthology.orchestra.run.smithi084.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-23T14:21:48.600 INFO:teuthology.orchestra.run.smithi084.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-23T14:21:48.602 INFO:teuthology.orchestra.run.smithi084.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-23T14:21:48.603 INFO:teuthology.orchestra.run.smithi084.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-23T14:21:48.604 INFO:teuthology.orchestra.run.smithi084.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-23T14:21:48.605 INFO:teuthology.orchestra.run.smithi084.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-23T14:21:48.607 INFO:teuthology.orchestra.run.smithi084.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-23T14:21:48.608 INFO:teuthology.orchestra.run.smithi084.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-23T14:21:48.610 INFO:teuthology.orchestra.run.smithi084.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-23T14:21:48.611 INFO:teuthology.orchestra.run.smithi084.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-23T14:21:48.623 INFO:teuthology.orchestra.run.smithi084.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-23T14:21:48.643 INFO:teuthology.orchestra.run.smithi084.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-23T14:21:48.644 INFO:teuthology.orchestra.run.smithi084.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-23T14:21:48.644 INFO:teuthology.orchestra.run.smithi084.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-23T14:21:54.052 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 13.2 MB in 2s (7,186 kB/s) 2024-04-23T14:21:55.321 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-04-23T14:21:55.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-23T14:21:55.401 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-04-23T14:21:55.600 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2024-04-23T14:21:55.601 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2024-04-23T14:21:55.767 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T14:21:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T14:21:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T14:21:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T14:21:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-23T14:21:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: python2.7-minimal 2024-04-23T14:21:55.768 INFO:teuthology.orchestra.run.smithi084.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T14:21:55.769 INFO:teuthology.orchestra.run.smithi084.stdout:The following additional packages will be installed: 2024-04-23T14:21:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-23T14:21:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-23T14:21:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-23T14:21:55.770 INFO:teuthology.orchestra.run.smithi084.stdout:Suggested packages: 2024-04-23T14:21:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-23T14:21:55.770 INFO:teuthology.orchestra.run.smithi084.stdout:Recommended packages: 2024-04-23T14:21:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: thermald 2024-04-23T14:21:55.807 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2024-04-23T14:21:55.807 INFO:teuthology.orchestra.run.smithi084.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-23T14:21:55.807 INFO:teuthology.orchestra.run.smithi084.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-23T14:21:55.808 INFO:teuthology.orchestra.run.smithi084.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-23T14:21:55.809 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages will be upgraded: 2024-04-23T14:21:55.809 INFO:teuthology.orchestra.run.smithi084.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-23T14:21:55.890 INFO:teuthology.orchestra.run.smithi084.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-23T14:21:55.890 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 113 MB of archives. 2024-04-23T14:21:55.890 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-23T14:21:55.890 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-23T14:21:56.493 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-23T14:21:56.652 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-23T14:21:57.898 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-23T14:21:57.898 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-23T14:21:57.898 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-23T14:21:58.173 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-23T14:21:58.247 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-23T14:21:58.666 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 113 MB in 2s (46.6 MB/s) 2024-04-23T14:21:58.988 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-23T14:22:01.102 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-23T14:22:01.107 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T14:22:01.392 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:03.500 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-23T14:22:03.522 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T14:22:03.617 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:04.064 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-23T14:22:04.084 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T14:22:04.122 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:10.409 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T14:22:10.545 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T14:22:10.902 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T14:22:11.015 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T14:22:11.293 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-23T14:22:11.303 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-23T14:22:11.343 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-23T14:22:17.560 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-23T14:22:17.596 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-23T14:22:17.634 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:20.273 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-23T14:22:20.395 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-23T14:22:20.737 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-23T14:22:20.842 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:20.943 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-23T14:22:21.080 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:22.408 INFO:teuthology.orchestra.run.smithi084.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-23T14:22:22.408 INFO:teuthology.orchestra.run.smithi084.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-23T14:22:22.408 INFO:teuthology.orchestra.run.smithi084.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-23T14:22:22.408 INFO:teuthology.orchestra.run.smithi084.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-23T14:22:22.564 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:24.384 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:25.809 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-23T14:22:25.935 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-23T14:22:26.061 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-23T14:22:26.131 INFO:teuthology.orchestra.run.smithi084.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-23T14:22:26.131 INFO:teuthology.orchestra.run.smithi084.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-23T14:22:40.600 INFO:teuthology.orchestra.run.smithi084.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-23T14:22:40.600 INFO:teuthology.orchestra.run.smithi084.stdout:Sourcing file `/etc/default/grub' 2024-04-23T14:22:40.625 INFO:teuthology.orchestra.run.smithi084.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-23T14:22:40.648 INFO:teuthology.orchestra.run.smithi084.stdout:Generating grub configuration file ... 2024-04-23T14:22:40.913 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-23T14:22:40.935 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-23T14:22:41.222 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-23T14:22:41.228 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-23T14:22:41.305 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-23T14:22:41.310 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-23T14:22:41.470 INFO:teuthology.orchestra.run.smithi084.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-23T14:22:41.471 INFO:teuthology.orchestra.run.smithi084.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-23T14:22:41.471 INFO:teuthology.orchestra.run.smithi084.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-23T14:22:41.516 INFO:teuthology.orchestra.run.smithi084.stdout:done 2024-04-23T14:22:41.967 DEBUG:teuthology.orchestra.run.smithi084:> dpkg -s linux-image-generic 2024-04-23T14:22:41.987 INFO:teuthology.orchestra.run.smithi084.stdout:Package: linux-image-generic 2024-04-23T14:22:41.987 INFO:teuthology.orchestra.run.smithi084.stdout:Status: install ok installed 2024-04-23T14:22:41.987 INFO:teuthology.orchestra.run.smithi084.stdout:Priority: optional 2024-04-23T14:22:41.987 INFO:teuthology.orchestra.run.smithi084.stdout:Section: kernel 2024-04-23T14:22:41.987 INFO:teuthology.orchestra.run.smithi084.stdout:Installed-Size: 21 2024-04-23T14:22:41.987 INFO:teuthology.orchestra.run.smithi084.stdout:Maintainer: Ubuntu Kernel Team 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: amd64 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout:Source: linux-meta 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout:Version: 5.15.0.105.102 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout:Recommends: thermald 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout:Description: Generic Linux kernel image 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: This package will always depend on the latest generic kernel image 2024-04-23T14:22:41.988 INFO:teuthology.orchestra.run.smithi084.stdout: available. 2024-04-23T14:22:41.988 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-23T14:22:41.988 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-23T14:22:41.989 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-23T14:22:41.989 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-23T14:22:41.989 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-04-23T14:22:41.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get clean 2024-04-23T14:22:42.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2024-04-23T14:22:42.310 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T14:22:42.310 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-23T14:22:42.323 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-23T14:22:42.344 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-23T14:22:43.701 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-04-23T14:22:43.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-23T14:22:43.771 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-04-23T14:22:43.948 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2024-04-23T14:22:43.949 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2024-04-23T14:22:44.109 INFO:teuthology.orchestra.run.smithi084.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-23T14:22:44.110 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T14:22:44.110 INFO:teuthology.orchestra.run.smithi084.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T14:22:44.110 INFO:teuthology.orchestra.run.smithi084.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T14:22:44.110 INFO:teuthology.orchestra.run.smithi084.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T14:22:44.110 INFO:teuthology.orchestra.run.smithi084.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-23T14:22:44.111 INFO:teuthology.orchestra.run.smithi084.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-23T14:22:44.111 INFO:teuthology.orchestra.run.smithi084.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T14:22:44.149 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-23T14:22:44.151 DEBUG:teuthology.orchestra.run.smithi084:> dpkg -s linux-image-generic 2024-04-23T14:22:44.209 INFO:teuthology.orchestra.run.smithi084.stdout:Package: linux-image-generic 2024-04-23T14:22:44.209 INFO:teuthology.orchestra.run.smithi084.stdout:Status: install ok installed 2024-04-23T14:22:44.209 INFO:teuthology.orchestra.run.smithi084.stdout:Priority: optional 2024-04-23T14:22:44.209 INFO:teuthology.orchestra.run.smithi084.stdout:Section: kernel 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Installed-Size: 21 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Maintainer: Ubuntu Kernel Team 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: amd64 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Source: linux-meta 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Version: 5.15.0.105.102 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Recommends: thermald 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout:Description: Generic Linux kernel image 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout: This package will always depend on the latest generic kernel image 2024-04-23T14:22:44.210 INFO:teuthology.orchestra.run.smithi084.stdout: available. 2024-04-23T14:22:44.211 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-04-23T14:22:44.258 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.l5UnMmqBcP 2024-04-23T14:22:44.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub/grub.cfg /tmp/tmp.l5UnMmqBcP 2024-04-23T14:22:44.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.l5UnMmqBcP 2024-04-23T14:22:44.436 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.l5UnMmqBcP is 10KB 2024-04-23T14:22:44.449 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.l5UnMmqBcP 2024-04-23T14:22:44.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-23T14:22:44.512 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:22:44.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-23T14:22:44.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-23T14:22:44.587 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-23T14:22:44.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo update-grub 2024-04-23T14:22:45.744 INFO:teuthology.orchestra.run.smithi084.stderr:Sourcing file `/etc/default/grub' 2024-04-23T14:22:45.745 INFO:teuthology.orchestra.run.smithi084.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-23T14:22:45.748 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-04-23T14:22:45.929 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-23T14:22:45.934 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-23T14:22:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-23T14:22:46.164 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-23T14:22:46.202 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-23T14:22:46.206 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-23T14:22:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-23T14:22:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-23T14:22:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-23T14:22:46.357 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-04-23T14:22:46.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-04-23T14:23:16.394 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-23T14:23:16.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-23T14:23:16.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:23:34.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-04-23T14:23:43.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-23T14:23:43.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:23:47.010 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-04-23T14:23:59.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-23T14:23:59.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:24:17.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-04-23T14:24:32.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-23T14:24:32.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:24:32.869 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-04-23T14:24:33.535 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-23T14:24:33.535 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-23T14:24:33.536 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-04-23T14:24:33.581 INFO:teuthology.orchestra.run.smithi084.stdout:5.15.0-105-generic 2024-04-23T14:24:33.581 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-23T14:24:33.582 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T14:24:33.582 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-23T14:24:34.582 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-23T14:24:34.583 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T14:24:34.644 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-04-23T14:24:34.667 DEBUG:teuthology.parallel:result is None 2024-04-23T14:24:34.667 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T14:24:34.678 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T14:24:34.678 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T14:24:34.683 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T14:24:34.702 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T14:24:34.718 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T14:24:34.718 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T14:24:34.745 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T14:24:34.756 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T14:24:34.756 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T14:24:34.804 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:24:34.812 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:24:34.814 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T14:24:34.823 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T14:24:34.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T14:24:34.869 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T14:24:34.882 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T14:24:34.882 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T14:24:34.917 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:24:34.966 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:24:35.014 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:24:35.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T14:24:35.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-04-23T14:24:35.158 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T14:24:35.166 INFO:teuthology.task.internal:Starting timer... 2024-04-23T14:24:35.167 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T14:24:35.250 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T14:24:35.262 DEBUG:teuthology.task.selinux:Excluding smithi084: OS 'ubuntu' does not support SELinux 2024-04-23T14:24:35.262 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T14:24:35.263 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-23T14:24:35.263 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T14:24:35.263 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T14:24:35.278 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T14:24:35.296 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T14:24:35.298 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T14:31:44.116 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com')] 2024-04-23T14:31:44.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-23T14:31:44.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:31:44.202 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-04-23T14:31:44.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-23T14:31:44.277 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T14:31:44.286 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T14:31:44.287 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T14:31:44.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Command line: ntpd -gq 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: ---------------------------------------------------- 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: ntp-4 is maintained by Network Time Foundation, 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: corporation. Support and training for ntp-4 are 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: available at https://www.nwtime.org/support 2024-04-23T14:31:44.340 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: ---------------------------------------------------- 2024-04-23T14:31:44.341 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: proto: precision = 0.061 usec (-24) 2024-04-23T14:31:44.341 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: basedate set to 2022-02-04 2024-04-23T14:31:44.341 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: gps base set to 2022-02-06 (week 2196) 2024-04-23T14:31:44.341 INFO:teuthology.orchestra.run.smithi084.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-23T14:31:44.342 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-23T14:31:44.342 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: restrict ::: KOD does nothing without LIMITED. 2024-04-23T14:31:44.342 INFO:teuthology.orchestra.run.smithi084.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-23T14:31:44.343 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Listen and drop on 0 v6wildcard [::]:123 2024-04-23T14:31:44.343 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-23T14:31:44.343 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Listen normally on 2 lo 127.0.0.1:123 2024-04-23T14:31:44.343 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Listen normally on 3 enp3s0f1 172.21.15.84:123 2024-04-23T14:31:44.343 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Listen normally on 4 lo [::1]:123 2024-04-23T14:31:44.343 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d751%5]:123 2024-04-23T14:31:44.343 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:44 ntpd[17207]: Listening on routing socket on fd #22 for interface updates 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:45.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:45 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:46.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:46.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:46.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:46.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:46.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:46.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:46 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:47.340 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:47 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:47.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:47 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:47.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:47 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:47.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:47 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:47.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:47 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:47.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:47 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:48.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:48.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:48.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:48.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:48.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:48 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:49.340 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:49 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:49.340 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:49 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:49.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:49 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:49.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:49 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:49.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:49 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:49.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:49 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:50.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:50.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:50.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:50.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:50.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:50.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:50.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:50.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:50.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:50.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:50.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:50.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:50 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:51.340 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:51 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:51 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:51 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:51 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:51 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:51 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:52.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:52.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:52.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:52.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:52.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-23T14:31:52.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: can't open /var/log/ntpstats/rawstats.20240423: Permission denied 2024-04-23T14:31:52.341 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-23T14:31:52.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: can't open /var/log/ntpstats/peerstats.20240423: Permission denied 2024-04-23T14:31:52.342 INFO:teuthology.orchestra.run.smithi084.stdout:23 Apr 14:31:52 ntpd[17207]: ntpd: time slew -0.001330 s 2024-04-23T14:31:52.342 INFO:teuthology.orchestra.run.smithi084.stdout:ntpd: time slew -0.001330s 2024-04-23T14:31:52.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-23T14:31:52.342 INFO:teuthology.orchestra.run.smithi084.stderr:23 Apr 14:31:52 ntpd[17207]: can't open /var/log/ntpstats/loopstats.20240423: Permission denied 2024-04-23T14:31:52.392 INFO:teuthology.orchestra.run.smithi084.stdout: remote refid st t when poll reach delay offset jitter 2024-04-23T14:31:52.392 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================== 2024-04-23T14:31:52.392 INFO:teuthology.orchestra.run.smithi084.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T14:31:52.392 INFO:teuthology.orchestra.run.smithi084.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T14:31:52.392 INFO:teuthology.orchestra.run.smithi084.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T14:31:52.392 INFO:teuthology.orchestra.run.smithi084.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T14:31:52.393 INFO:teuthology.run_tasks:Running task install... 2024-04-23T14:31:52.402 DEBUG:teuthology.task.install:project ceph 2024-04-23T14:31:52.402 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}} 2024-04-23T14:31:52.402 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'} 2024-04-23T14:31:52.403 INFO:teuthology.task.install:Using flavor: default 2024-04-23T14:31:52.408 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T14:31:52.408 INFO:teuthology.task.install:extra packages: [] 2024-04-23T14:31:52.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-key list | grep Ceph 2024-04-23T14:31:52.490 INFO:teuthology.orchestra.run.smithi084.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-23T14:31:52.516 INFO:teuthology.orchestra.run.smithi084.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-23T14:31:52.516 INFO:teuthology.orchestra.run.smithi084.stdout:uid [ unknown] Ceph.com (release key) 2024-04-23T14:31:52.517 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-23T14:31:52.517 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T14:31:52.838 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default/ 2024-04-23T14:31:52.838 INFO:teuthology.task.install.deb:Package version is 19.0.0-3211-g5d349943-1jammy 2024-04-23T14:31:53.066 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:31:53.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-23T14:31:53.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2024-04-23T14:31:53.313 INFO:teuthology.orchestra.run.smithi084.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy InRelease 2024-04-23T14:31:53.324 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-23T14:31:53.335 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-23T14:31:53.358 INFO:teuthology.orchestra.run.smithi084.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-23T14:31:53.366 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-23T14:31:53.380 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2024-04-23T14:31:53.426 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-23T14:31:53.448 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-23T14:31:53.532 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-23T14:31:53.678 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 29.2 kB in 0s (69.4 kB/s) 2024-04-23T14:31:54.615 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-04-23T14:31:54.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3211-g5d349943-1jammy cephadm=19.0.0-3211-g5d349943-1jammy ceph-mds=19.0.0-3211-g5d349943-1jammy ceph-mgr=19.0.0-3211-g5d349943-1jammy ceph-common=19.0.0-3211-g5d349943-1jammy ceph-fuse=19.0.0-3211-g5d349943-1jammy ceph-test=19.0.0-3211-g5d349943-1jammy ceph-volume=19.0.0-3211-g5d349943-1jammy radosgw=19.0.0-3211-g5d349943-1jammy python3-rados=19.0.0-3211-g5d349943-1jammy python3-rgw=19.0.0-3211-g5d349943-1jammy python3-cephfs=19.0.0-3211-g5d349943-1jammy python3-rbd=19.0.0-3211-g5d349943-1jammy libcephfs2=19.0.0-3211-g5d349943-1jammy libcephfs-dev=19.0.0-3211-g5d349943-1jammy librados2=19.0.0-3211-g5d349943-1jammy librbd1=19.0.0-3211-g5d349943-1jammy rbd-fuse=19.0.0-3211-g5d349943-1jammy 2024-04-23T14:31:54.693 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2024-04-23T14:31:54.901 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2024-04-23T14:31:54.902 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2024-04-23T14:31:55.095 INFO:teuthology.orchestra.run.smithi084.stdout:The following packages were automatically installed and are no longer required: 2024-04-23T14:31:55.095 INFO:teuthology.orchestra.run.smithi084.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-23T14:31:55.095 INFO:teuthology.orchestra.run.smithi084.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-23T14:31:55.095 INFO:teuthology.orchestra.run.smithi084.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-23T14:31:55.095 INFO:teuthology.orchestra.run.smithi084.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-23T14:31:55.096 INFO:teuthology.orchestra.run.smithi084.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-23T14:31:55.096 INFO:teuthology.orchestra.run.smithi084.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-23T14:31:55.096 INFO:teuthology.orchestra.run.smithi084.stdout:The following additional packages will be installed: 2024-04-23T14:31:55.096 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-23T14:31:55.096 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-23T14:31:55.097 INFO:teuthology.orchestra.run.smithi084.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-23T14:31:55.097 INFO:teuthology.orchestra.run.smithi084.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-23T14:31:55.098 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-23T14:31:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet zip 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout:Suggested packages: 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout:Recommended packages: 2024-04-23T14:31:55.100 INFO:teuthology.orchestra.run.smithi084.stdout: btrfs-tools 2024-04-23T14:31:55.204 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2024-04-23T14:31:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-23T14:31:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-23T14:31:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-23T14:31:55.206 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-23T14:31:55.207 INFO:teuthology.orchestra.run.smithi084.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-23T14:31:55.395 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-23T14:31:55.395 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 181 MB of archives. 2024-04-23T14:31:55.395 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 828 MB of additional disk space will be used. 2024-04-23T14:31:55.395 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-23T14:31:55.586 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3211-g5d349943-1jammy [3,554 kB] 2024-04-23T14:31:55.821 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-23T14:31:55.832 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-23T14:31:55.914 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-23T14:31:56.144 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-23T14:31:56.155 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-23T14:31:56.185 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-23T14:31:56.194 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-23T14:31:56.195 INFO:teuthology.orchestra.run.smithi084.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-23T14:31:56.196 INFO:teuthology.orchestra.run.smithi084.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-23T14:31:56.215 INFO:teuthology.orchestra.run.smithi084.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-23T14:31:56.233 INFO:teuthology.orchestra.run.smithi084.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3211-g5d349943-1jammy [3,223 kB] 2024-04-23T14:31:56.338 INFO:teuthology.orchestra.run.smithi084.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3211-g5d349943-1jammy [712 kB] 2024-04-23T14:31:56.361 INFO:teuthology.orchestra.run.smithi084.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3211-g5d349943-1jammy [357 kB] 2024-04-23T14:31:56.372 INFO:teuthology.orchestra.run.smithi084.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3211-g5d349943-1jammy [32.8 kB] 2024-04-23T14:31:56.373 INFO:teuthology.orchestra.run.smithi084.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3211-g5d349943-1jammy [180 kB] 2024-04-23T14:31:56.379 INFO:teuthology.orchestra.run.smithi084.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3211-g5d349943-1jammy [66.9 kB] 2024-04-23T14:31:56.381 INFO:teuthology.orchestra.run.smithi084.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3211-g5d349943-1jammy [330 kB] 2024-04-23T14:31:56.384 INFO:teuthology.orchestra.run.smithi084.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-23T14:31:56.392 INFO:teuthology.orchestra.run.smithi084.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3211-g5d349943-1jammy [6,894 kB] 2024-04-23T14:31:56.620 INFO:teuthology.orchestra.run.smithi084.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3211-g5d349943-1jammy [112 kB] 2024-04-23T14:31:56.623 INFO:teuthology.orchestra.run.smithi084.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3211-g5d349943-1jammy [466 kB] 2024-04-23T14:31:56.638 INFO:teuthology.orchestra.run.smithi084.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3211-g5d349943-1jammy [25.8 MB] 2024-04-23T14:31:56.785 INFO:teuthology.orchestra.run.smithi084.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-23T14:31:56.820 INFO:teuthology.orchestra.run.smithi084.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-23T14:31:56.822 INFO:teuthology.orchestra.run.smithi084.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-23T14:31:56.902 INFO:teuthology.orchestra.run.smithi084.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-23T14:31:56.954 INFO:teuthology.orchestra.run.smithi084.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-23T14:31:56.975 INFO:teuthology.orchestra.run.smithi084.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-23T14:31:56.981 INFO:teuthology.orchestra.run.smithi084.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-23T14:31:56.983 INFO:teuthology.orchestra.run.smithi084.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-23T14:31:56.994 INFO:teuthology.orchestra.run.smithi084.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-23T14:31:56.995 INFO:teuthology.orchestra.run.smithi084.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-23T14:31:57.160 INFO:teuthology.orchestra.run.smithi084.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-23T14:31:57.243 INFO:teuthology.orchestra.run.smithi084.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-23T14:31:57.272 INFO:teuthology.orchestra.run.smithi084.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-23T14:31:57.289 INFO:teuthology.orchestra.run.smithi084.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-23T14:31:57.333 INFO:teuthology.orchestra.run.smithi084.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-23T14:31:57.581 INFO:teuthology.orchestra.run.smithi084.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-23T14:31:57.582 INFO:teuthology.orchestra.run.smithi084.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-23T14:31:57.582 INFO:teuthology.orchestra.run.smithi084.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3211-g5d349943-1jammy [5,092 kB] 2024-04-23T14:31:57.583 INFO:teuthology.orchestra.run.smithi084.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-23T14:31:57.584 INFO:teuthology.orchestra.run.smithi084.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-23T14:31:57.596 INFO:teuthology.orchestra.run.smithi084.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-23T14:31:57.600 INFO:teuthology.orchestra.run.smithi084.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-23T14:31:57.694 INFO:teuthology.orchestra.run.smithi084.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3211-g5d349943-1jammy [244 kB] 2024-04-23T14:31:57.698 INFO:teuthology.orchestra.run.smithi084.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3211-g5d349943-1jammy [124 kB] 2024-04-23T14:31:57.702 INFO:teuthology.orchestra.run.smithi084.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3211-g5d349943-1jammy [1,527 kB] 2024-04-23T14:31:57.765 INFO:teuthology.orchestra.run.smithi084.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3211-g5d349943-1jammy [6,258 kB] 2024-04-23T14:31:57.767 INFO:teuthology.orchestra.run.smithi084.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-23T14:31:57.972 INFO:teuthology.orchestra.run.smithi084.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3211-g5d349943-1jammy [22.8 MB] 2024-04-23T14:31:58.101 INFO:teuthology.orchestra.run.smithi084.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-23T14:31:58.133 INFO:teuthology.orchestra.run.smithi084.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-23T14:31:58.144 INFO:teuthology.orchestra.run.smithi084.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-23T14:31:58.346 INFO:teuthology.orchestra.run.smithi084.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-23T14:31:58.347 INFO:teuthology.orchestra.run.smithi084.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-23T14:31:58.353 INFO:teuthology.orchestra.run.smithi084.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-23T14:31:58.360 INFO:teuthology.orchestra.run.smithi084.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-23T14:31:58.375 INFO:teuthology.orchestra.run.smithi084.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-23T14:31:58.396 INFO:teuthology.orchestra.run.smithi084.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-23T14:31:58.402 INFO:teuthology.orchestra.run.smithi084.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-23T14:31:58.577 INFO:teuthology.orchestra.run.smithi084.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-23T14:31:58.802 INFO:teuthology.orchestra.run.smithi084.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3211-g5d349943-1jammy [14.0 kB] 2024-04-23T14:31:58.803 INFO:teuthology.orchestra.run.smithi084.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3211-g5d349943-1jammy [870 kB] 2024-04-23T14:31:58.820 INFO:teuthology.orchestra.run.smithi084.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3211-g5d349943-1jammy [2,436 kB] 2024-04-23T14:31:58.860 INFO:teuthology.orchestra.run.smithi084.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3211-g5d349943-1jammy [791 kB] 2024-04-23T14:31:58.869 INFO:teuthology.orchestra.run.smithi084.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3211-g5d349943-1jammy [148 kB] 2024-04-23T14:31:58.871 INFO:teuthology.orchestra.run.smithi084.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3211-g5d349943-1jammy [2,297 kB] 2024-04-23T14:31:58.883 INFO:teuthology.orchestra.run.smithi084.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-23T14:31:59.007 INFO:teuthology.orchestra.run.smithi084.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3211-g5d349943-1jammy [8,625 kB] 2024-04-23T14:31:59.062 INFO:teuthology.orchestra.run.smithi084.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-23T14:31:59.076 INFO:teuthology.orchestra.run.smithi084.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-23T14:31:59.096 INFO:teuthology.orchestra.run.smithi084.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-23T14:31:59.161 INFO:teuthology.orchestra.run.smithi084.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-23T14:31:59.289 INFO:teuthology.orchestra.run.smithi084.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3211-g5d349943-1jammy [14.2 kB] 2024-04-23T14:31:59.289 INFO:teuthology.orchestra.run.smithi084.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3211-g5d349943-1jammy [51.6 MB] 2024-04-23T14:31:59.449 INFO:teuthology.orchestra.run.smithi084.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-23T14:31:59.451 INFO:teuthology.orchestra.run.smithi084.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-23T14:31:59.454 INFO:teuthology.orchestra.run.smithi084.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-23T14:31:59.454 INFO:teuthology.orchestra.run.smithi084.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-23T14:31:59.457 INFO:teuthology.orchestra.run.smithi084.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-23T14:31:59.675 INFO:teuthology.orchestra.run.smithi084.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-23T14:32:00.106 INFO:teuthology.orchestra.run.smithi084.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-23T14:32:00.112 INFO:teuthology.orchestra.run.smithi084.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-23T14:32:00.402 INFO:teuthology.orchestra.run.smithi084.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-23T14:32:00.403 INFO:teuthology.orchestra.run.smithi084.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-23T14:32:00.405 INFO:teuthology.orchestra.run.smithi084.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-23T14:32:00.406 INFO:teuthology.orchestra.run.smithi084.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-23T14:32:00.408 INFO:teuthology.orchestra.run.smithi084.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-23T14:32:00.410 INFO:teuthology.orchestra.run.smithi084.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-23T14:32:00.411 INFO:teuthology.orchestra.run.smithi084.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-23T14:32:00.413 INFO:teuthology.orchestra.run.smithi084.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-23T14:32:00.594 INFO:teuthology.orchestra.run.smithi084.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-23T14:32:01.007 INFO:teuthology.orchestra.run.smithi084.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-23T14:32:01.053 INFO:teuthology.orchestra.run.smithi084.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-23T14:32:01.088 INFO:teuthology.orchestra.run.smithi084.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-23T14:32:01.185 INFO:teuthology.orchestra.run.smithi084.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-23T14:32:01.225 INFO:teuthology.orchestra.run.smithi084.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-23T14:32:01.236 INFO:teuthology.orchestra.run.smithi084.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-23T14:32:01.242 INFO:teuthology.orchestra.run.smithi084.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-23T14:32:01.289 INFO:teuthology.orchestra.run.smithi084.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-23T14:32:01.292 INFO:teuthology.orchestra.run.smithi084.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-23T14:32:01.313 INFO:teuthology.orchestra.run.smithi084.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-23T14:32:01.478 INFO:teuthology.orchestra.run.smithi084.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-23T14:32:01.824 INFO:teuthology.orchestra.run.smithi084.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-23T14:32:01.880 INFO:teuthology.orchestra.run.smithi084.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-23T14:32:01.880 INFO:teuthology.orchestra.run.smithi084.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3211-g5d349943-1jammy [124 kB] 2024-04-23T14:32:01.883 INFO:teuthology.orchestra.run.smithi084.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3211-g5d349943-1jammy [38.9 kB] 2024-04-23T14:32:01.884 INFO:teuthology.orchestra.run.smithi084.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3211-g5d349943-1jammy [13.6 MB] 2024-04-23T14:32:01.995 INFO:teuthology.orchestra.run.smithi084.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-23T14:32:02.024 INFO:teuthology.orchestra.run.smithi084.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-23T14:32:02.182 INFO:teuthology.orchestra.run.smithi084.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-23T14:32:02.185 INFO:teuthology.orchestra.run.smithi084.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-23T14:32:02.187 INFO:teuthology.orchestra.run.smithi084.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-23T14:32:02.209 INFO:teuthology.orchestra.run.smithi084.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-23T14:32:02.214 INFO:teuthology.orchestra.run.smithi084.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-23T14:32:02.427 INFO:teuthology.orchestra.run.smithi084.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2024-04-05-0825/5d349943c59c9485df060d6adb0594f3940ec0eb/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3211-g5d349943-1jammy [91.6 kB] 2024-04-23T14:32:02.991 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 181 MB in 7s (25.1 MB/s) 2024-04-23T14:32:03.134 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-23T14:32:03.233 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-23T14:32:03.239 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-23T14:32:03.324 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-23T14:32:03.669 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-23T14:32:03.695 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-23T14:32:03.736 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-23T14:32:04.056 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-23T14:32:04.082 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-23T14:32:04.122 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-23T14:32:04.534 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-23T14:32:04.560 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T14:32:04.609 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T14:32:05.021 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-23T14:32:05.047 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T14:32:05.087 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T14:32:05.415 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-23T14:32:05.441 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-23T14:32:05.473 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T14:32:05.818 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-23T14:32:05.844 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-23T14:32:05.884 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-23T14:32:06.198 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librados2. 2024-04-23T14:32:06.213 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../007-librados2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:06.247 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librados2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:06.692 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librbd1. 2024-04-23T14:32:06.719 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../008-librbd1_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:06.758 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librbd1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:07.195 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libcephfs2. 2024-04-23T14:32:07.221 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:07.261 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libcephfs2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:07.547 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rados. 2024-04-23T14:32:07.562 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:07.597 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rados (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:07.891 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-23T14:32:07.906 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:07.941 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-ceph-argparse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:08.168 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cephfs. 2024-04-23T14:32:08.184 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:08.227 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cephfs (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:08.563 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-23T14:32:08.589 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:08.629 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:08.958 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-23T14:32:08.984 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-23T14:32:09.024 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-23T14:32:09.302 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-prettytable. 2024-04-23T14:32:09.317 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-23T14:32:09.352 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-23T14:32:09.554 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rbd. 2024-04-23T14:32:09.567 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:09.604 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rbd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:09.974 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-23T14:32:10.000 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-23T14:32:10.040 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-23T14:32:10.326 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-23T14:32:10.342 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-23T14:32:10.384 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-23T14:32:10.671 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-23T14:32:10.686 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-23T14:32:10.720 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-23T14:32:11.041 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua5.1. 2024-04-23T14:32:11.067 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-23T14:32:11.107 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-23T14:32:11.427 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua-any. 2024-04-23T14:32:11.442 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-23T14:32:11.485 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-23T14:32:11.737 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package zip. 2024-04-23T14:32:11.763 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-23T14:32:11.804 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking zip (3.0-12build2) ... 2024-04-23T14:32:12.082 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package unzip. 2024-04-23T14:32:12.098 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-23T14:32:12.132 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-23T14:32:12.528 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package luarocks. 2024-04-23T14:32:12.554 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-23T14:32:12.594 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-23T14:32:13.219 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package librgw2. 2024-04-23T14:32:13.245 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../025-librgw2_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:13.310 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking librgw2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:13.863 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rgw. 2024-04-23T14:32:13.889 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:13.929 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:14.224 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-23T14:32:14.239 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-23T14:32:14.273 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-23T14:32:14.552 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libradosstriper1. 2024-04-23T14:32:14.578 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:14.618 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libradosstriper1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:15.047 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-common. 2024-04-23T14:32:15.073 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:15.121 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:16.476 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-base. 2024-04-23T14:32:16.502 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:16.544 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-base (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:17.097 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-23T14:32:17.124 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-23T14:32:17.164 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-23T14:32:17.466 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cheroot. 2024-04-23T14:32:17.493 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-23T14:32:17.533 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-23T14:32:17.861 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-23T14:32:17.888 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-23T14:32:17.927 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-23T14:32:18.239 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-23T14:32:18.266 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-23T14:32:18.305 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-23T14:32:18.541 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-23T14:32:18.568 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-23T14:32:18.608 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-23T14:32:18.911 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-tempora. 2024-04-23T14:32:18.937 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-23T14:32:18.977 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-23T14:32:19.222 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-portend. 2024-04-23T14:32:19.248 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-23T14:32:19.288 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-23T14:32:19.558 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-23T14:32:19.585 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-23T14:32:19.616 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-23T14:32:19.919 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-23T14:32:19.945 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-23T14:32:19.985 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-23T14:32:20.314 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-natsort. 2024-04-23T14:32:20.330 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-23T14:32:20.364 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-23T14:32:20.659 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-logutils. 2024-04-23T14:32:20.685 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-23T14:32:20.725 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-23T14:32:21.011 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-23T14:32:21.038 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-23T14:32:21.078 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-23T14:32:21.389 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-mako. 2024-04-23T14:32:21.415 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-23T14:32:21.456 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-23T14:32:21.775 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-23T14:32:21.802 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-23T14:32:21.842 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-23T14:32:22.078 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-23T14:32:22.105 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-23T14:32:22.144 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-23T14:32:22.456 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-webob. 2024-04-23T14:32:22.483 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-23T14:32:22.522 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-23T14:32:22.875 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-waitress. 2024-04-23T14:32:22.902 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-23T14:32:22.945 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-23T14:32:23.228 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-tempita. 2024-04-23T14:32:23.255 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-23T14:32:23.461 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-23T14:32:23.781 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-paste. 2024-04-23T14:32:23.807 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-23T14:32:23.847 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-23T14:32:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-23T14:32:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-23T14:32:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-23T14:32:24.520 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-23T14:32:24.546 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-23T14:32:24.578 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-23T14:32:24.856 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-webtest. 2024-04-23T14:32:24.882 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-23T14:32:24.922 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-23T14:32:25.209 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pecan. 2024-04-23T14:32:25.235 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-23T14:32:25.275 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-23T14:32:25.647 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-certifi. 2024-04-23T14:32:25.674 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-23T14:32:25.713 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-23T14:32:26.041 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-idna. 2024-04-23T14:32:26.067 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-23T14:32:26.107 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-23T14:32:26.427 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-urllib3. 2024-04-23T14:32:26.454 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-23T14:32:26.493 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-23T14:32:26.805 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-requests. 2024-04-23T14:32:26.832 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-23T14:32:26.871 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-23T14:32:27.150 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-23T14:32:27.165 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-23T14:32:27.200 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-23T14:32:27.494 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-23T14:32:27.510 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:27.552 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:27.941 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-23T14:32:27.967 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:28.007 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:28.326 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-23T14:32:28.353 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-23T14:32:28.635 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-23T14:32:29.421 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-openssl. 2024-04-23T14:32:29.436 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-23T14:32:29.471 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-23T14:32:29.783 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr. 2024-04-23T14:32:29.810 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:29.849 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:30.227 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mon. 2024-04-23T14:32:30.254 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:30.293 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mon (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:30.830 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-osd. 2024-04-23T14:32:30.857 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:30.888 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-osd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:31.925 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph. 2024-04-23T14:32:31.951 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../066-ceph_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:31.991 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:32.278 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-fuse. 2024-04-23T14:32:32.305 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:32.344 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:32.681 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mds. 2024-04-23T14:32:32.707 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:32.747 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mds (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:33.150 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package cephadm. 2024-04-23T14:32:33.177 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../069-cephadm_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:33.216 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:33.678 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-23T14:32:33.705 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-23T14:32:33.802 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-23T14:32:34.138 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-23T14:32:34.153 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-23T14:32:34.196 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-23T14:32:35.209 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-babel. 2024-04-23T14:32:35.226 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-23T14:32:35.259 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-23T14:32:35.554 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-jinja2. 2024-04-23T14:32:35.581 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-23T14:32:35.620 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-23T14:32:35.949 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-23T14:32:35.976 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:36.015 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:36.310 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-23T14:32:36.337 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-23T14:32:36.376 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-23T14:32:36.679 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-routes. 2024-04-23T14:32:36.706 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-23T14:32:36.745 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-23T14:32:37.099 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-23T14:32:37.126 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:37.165 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:38.429 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-23T14:32:38.456 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-23T14:32:38.495 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-23T14:32:39.007 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-joblib. 2024-04-23T14:32:39.034 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-23T14:32:39.073 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-23T14:32:39.427 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-23T14:32:39.454 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-23T14:32:39.493 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-23T14:32:39.788 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-sklearn. 2024-04-23T14:32:39.814 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-23T14:32:39.854 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-23T14:32:40.512 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-23T14:32:40.540 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:40.656 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:41.593 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-cachetools. 2024-04-23T14:32:41.620 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-23T14:32:41.659 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-23T14:32:41.937 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-23T14:32:41.953 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-23T14:32:41.995 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-23T14:32:42.274 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-23T14:32:42.301 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-23T14:32:42.332 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-23T14:32:42.601 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-rsa. 2024-04-23T14:32:42.629 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-23T14:32:42.668 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-23T14:32:42.988 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-google-auth. 2024-04-23T14:32:43.015 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-23T14:32:43.057 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-23T14:32:43.393 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-23T14:32:43.420 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-23T14:32:43.459 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-23T14:32:43.729 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-websocket. 2024-04-23T14:32:43.745 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-23T14:32:43.779 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-23T14:32:44.100 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-23T14:32:44.127 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-23T14:32:44.166 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-23T14:32:44.838 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-23T14:32:44.854 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:44.896 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:45.199 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-23T14:32:45.226 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-23T14:32:45.257 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-23T14:32:45.627 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-23T14:32:45.654 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-23T14:32:45.693 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-23T14:32:45.962 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package jq. 2024-04-23T14:32:45.989 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-23T14:32:46.029 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-23T14:32:46.282 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package socat. 2024-04-23T14:32:46.309 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-23T14:32:46.348 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-23T14:32:46.643 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package xmlstarlet. 2024-04-23T14:32:46.658 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-23T14:32:46.692 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-23T14:32:47.030 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-test. 2024-04-23T14:32:47.046 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:47.088 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-test (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:49.651 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ceph-volume. 2024-04-23T14:32:49.679 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3211-g5d349943-1jammy_all.deb ... 2024-04-23T14:32:49.717 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ceph-volume (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:50.013 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-23T14:32:50.041 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:50.079 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libcephfs-dev (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:50.424 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-23T14:32:50.451 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-23T14:32:50.490 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-23T14:32:50.803 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-23T14:32:50.831 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-23T14:32:50.861 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-23T14:32:51.224 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package nvme-cli. 2024-04-23T14:32:51.251 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-23T14:32:51.379 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-23T14:32:51.720 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package pkg-config. 2024-04-23T14:32:51.748 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-23T14:32:51.786 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-23T14:32:52.064 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-23T14:32:52.092 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-23T14:32:52.130 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-23T14:32:52.542 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-23T14:32:52.570 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-23T14:32:52.609 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-23T14:32:52.936 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pastescript. 2024-04-23T14:32:52.964 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-23T14:32:53.003 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-23T14:32:53.298 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pluggy. 2024-04-23T14:32:53.322 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-23T14:32:53.364 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-23T14:32:53.651 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-psutil. 2024-04-23T14:32:53.678 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-23T14:32:53.717 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-23T14:32:54.037 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-py. 2024-04-23T14:32:54.064 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-23T14:32:54.103 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-23T14:32:54.373 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pygments. 2024-04-23T14:32:54.386 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-23T14:32:54.414 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-23T14:32:55.102 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-23T14:32:55.130 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-23T14:32:55.168 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-23T14:32:55.479 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-toml. 2024-04-23T14:32:55.507 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-23T14:32:55.546 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-23T14:32:55.832 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-pytest. 2024-04-23T14:32:55.859 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-23T14:32:55.898 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-23T14:32:56.235 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package python3-simplejson. 2024-04-23T14:32:56.263 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-23T14:32:56.326 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-23T14:32:56.596 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-23T14:32:56.624 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-23T14:32:56.662 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-23T14:32:57.408 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package radosgw. 2024-04-23T14:32:57.436 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../116-radosgw_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:57.482 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking radosgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:58.344 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package rbd-fuse. 2024-04-23T14:32:58.372 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3211-g5d349943-1jammy_amd64.deb ... 2024-04-23T14:32:58.410 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking rbd-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:32:58.714 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-23T14:32:58.965 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-23T14:32:59.091 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-23T14:32:59.343 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-23T14:32:59.694 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-23T14:33:00.108 INFO:teuthology.orchestra.run.smithi084.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-23T14:33:00.176 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:00.667 INFO:teuthology.orchestra.run.smithi084.stdout:Adding system user cephadm....done 2024-04-23T14:33:00.807 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-23T14:33:01.075 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-23T14:33:01.335 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-23T14:33:01.436 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-23T14:33:01.687 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-23T14:33:01.946 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-23T14:33:02.072 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-23T14:33:02.416 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-23T14:33:02.768 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-23T14:33:03.044 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-23T14:33:03.304 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-23T14:33:03.588 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-23T14:33:03.847 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-23T14:33:04.099 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-23T14:33:04.349 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-ceph-argparse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:04.601 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-23T14:33:04.701 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-23T14:33:04.802 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-23T14:33:04.928 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-23T14:33:05.179 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-23T14:33:05.569 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-23T14:33:05.611 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-23T14:33:05.674 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-23T14:33:05.800 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-23T14:33:06.160 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-23T14:33:06.278 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-23T14:33:06.554 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-23T14:33:06.813 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-23T14:33:07.123 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up zip (3.0-12build2) ... 2024-04-23T14:33:07.232 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-23T14:33:07.901 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-23T14:33:08.151 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-23T14:33:08.277 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-23T14:33:08.403 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-23T14:33:08.696 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-23T14:33:09.089 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-23T14:33:09.332 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-23T14:33:09.716 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-23T14:33:10.018 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-23T14:33:10.469 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-23T14:33:10.928 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-23T14:33:11.054 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:11.364 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-idna (3.3-1) ... 2024-04-23T14:33:11.674 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-23T14:33:12.862 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-23T14:33:13.114 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T14:33:13.248 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-23T14:33:13.499 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-23T14:33:13.625 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-23T14:33:13.751 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-23T14:33:14.060 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-23T14:33:14.312 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-23T14:33:14.562 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T14:33:14.697 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-23T14:33:15.007 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-23T14:33:15.241 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-23T14:33:15.484 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-23T14:33:15.762 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-23T14:33:16.011 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-23T14:33:16.270 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-23T14:33:16.405 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-23T14:33:16.673 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-23T14:33:16.815 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-23T14:33:17.091 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-23T14:33:17.392 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-23T14:33:17.643 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-23T14:33:17.769 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-23T14:33:17.912 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-23T14:33:18.001 INFO:teuthology.orchestra.run.smithi084.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-23T14:33:18.239 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-23T14:33:18.365 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-23T14:33:18.733 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-23T14:33:18.984 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-23T14:33:19.110 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-23T14:33:19.369 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-23T14:33:19.495 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-23T14:33:19.830 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-23T14:33:20.164 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-23T14:33:20.440 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-23T14:33:20.566 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-23T14:33:20.847 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-23T14:33:21.276 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-23T14:33:21.638 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-23T14:33:22.007 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-23T14:33:22.300 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-23T14:33:22.637 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-23T14:33:22.761 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librados2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:22.887 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:23.014 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-23T14:33:23.300 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libcephfs2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:23.409 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libradosstriper1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:23.535 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-23T14:33:23.820 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librbd1 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:23.929 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-modules-core (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:24.039 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-23T14:33:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-23T14:33:24.699 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libcephfs-dev (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:24.824 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rados (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:24.971 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up librgw2 (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:25.101 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-23T14:33:26.339 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rbd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:26.456 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up rbd-fuse (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:26.582 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-rgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:26.708 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up python3-cephfs (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:26.834 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-common (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:27.402 INFO:teuthology.orchestra.run.smithi084.stdout:Adding group ceph....done 2024-04-23T14:33:27.763 INFO:teuthology.orchestra.run.smithi084.stdout:Adding system user ceph....done 2024-04-23T14:33:27.955 INFO:teuthology.orchestra.run.smithi084.stdout:Setting system user ceph properties....done 2024-04-23T14:33:27.974 INFO:teuthology.orchestra.run.smithi084.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-23T14:33:28.106 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-23T14:33:28.414 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-23T14:33:28.886 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-test (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:29.004 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up radosgw (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:29.448 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:33:29.449 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:33:29.889 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-base (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:30.211 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-23T14:33:30.641 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mds (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:30.862 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-23T14:33:30.862 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-23T14:33:31.317 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:31.560 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-23T14:33:31.561 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-23T14:33:32.052 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-osd (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:32.363 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-23T14:33:32.363 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-23T14:33:32.872 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:32.973 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:33.099 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mon (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-23T14:33:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-23T14:33:33.800 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-cephadm (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:33.951 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:34.077 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-mgr-dashboard (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:34.220 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ceph-volume (19.0.0-3211-g5d349943-1jammy) ... 2024-04-23T14:33:34.572 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-23T14:33:34.665 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-23T14:33:34.809 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-23T14:33:35.942 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-23T14:33:36.652 INFO:teuthology.orchestra.run.smithi084.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-23T14:33:36.658 DEBUG:teuthology.parallel:result is None 2024-04-23T14:33:36.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T14:33:36.898 DEBUG:teuthology.orchestra.run.smithi084:> dpkg-query -W -f '${Version}' ceph 2024-04-23T14:33:36.916 INFO:teuthology.orchestra.run.smithi084.stdout:19.0.0-3211-g5d349943-1jammy 2024-04-23T14:33:36.916 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3211-g5d349943-1jammy 2024-04-23T14:33:36.916 INFO:teuthology.task.install:The correct ceph version 19.0.0-3211-g5d349943-1jammy is installed. 2024-04-23T14:33:36.918 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T14:33:36.918 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:36.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T14:33:36.973 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T14:33:36.973 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:36.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T14:33:37.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T14:33:37.084 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T14:33:37.084 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:37.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T14:33:37.140 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T14:33:37.196 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T14:33:37.197 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:37.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T14:33:37.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T14:33:37.308 INFO:teuthology.run_tasks:Running task ceph... 2024-04-23T14:33:37.387 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-23T14:33:37.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /var/log/ceph 2024-04-23T14:33:37.400 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-23T14:33:37.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-23T14:33:37.456 INFO:tasks.ceph:Creating extra log directories... 2024-04-23T14:33:37.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-23T14:33:37.520 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-23T14:33:37.521 INFO:tasks.ceph:config {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-23T14:33:37.521 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603', 'branch': 'wip-yuri3-testing-2024-04-05-0825', 'description': 'rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7669603', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2024-04-05-0825'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}, 'workunit': {'branch': 'wip-yuri3-testing-2024-04-05-0825', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0']], 'seed': 3753, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2024-04-05-0825', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'targets': {'smithi084.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIAH+D4TracU+auAUik3ADNJF4fsvqun5e8LmdeYNNZ8K8LVqzYlnMA7F/LeegIz1oECQ2fYnnachO48W8CKcI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['rados/test_health_warnings.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-23_14:14:08', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-23T14:33:37.521 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-23T14:33:37.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/run/ceph 2024-04-23T14:33:37.616 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:37.616 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T14:33:37.665 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T14:33:37.665 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-04-23T14:33:37.713 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T14:33:37.713 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:33:37.713 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-23T14:33:37.713 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:33:37.713 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-23 14:28:29.018889804 +0000 2024-04-23T14:33:37.713 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-23 14:28:29.014889840 +0000 2024-04-23T14:33:37.714 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-23 14:28:29.014889840 +0000 2024-04-23T14:33:37.714 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-04-23T14:33:37.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T14:33:37.767 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-23T14:33:37.767 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-23T14:33:37.767 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000337826 s, 1.5 MB/s 2024-04-23T14:33:37.768 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T14:33:37.819 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-23 14:28:29.822882622 +0000 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-23 14:28:29.550885051 +0000 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-23 14:28:29.550885051 +0000 2024-04-23T14:33:37.865 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-04-23T14:33:37.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T14:33:37.919 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-23T14:33:37.919 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-23T14:33:37.919 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000367279 s, 1.4 MB/s 2024-04-23T14:33:37.920 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T14:33:37.972 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-23 14:28:30.330878085 +0000 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-23 14:28:30.042880657 +0000 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-23 14:28:30.042880657 +0000 2024-04-23T14:33:38.021 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-04-23T14:33:38.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T14:33:38.075 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-23T14:33:38.075 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-23T14:33:38.075 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000445943 s, 1.1 MB/s 2024-04-23T14:33:38.076 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T14:33:38.126 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-04-23T14:33:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T14:33:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:33:38.174 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-04-23T14:33:38.174 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:33:38.174 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-23 14:28:30.810873797 +0000 2024-04-23T14:33:38.175 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-23 14:28:30.542876190 +0000 2024-04-23T14:33:38.175 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-23 14:28:30.542876190 +0000 2024-04-23T14:33:38.175 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-04-23T14:33:38.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T14:33:38.226 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-23T14:33:38.226 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-23T14:33:38.227 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000378101 s, 1.4 MB/s 2024-04-23T14:33:38.228 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T14:33:38.278 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:38.278 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-04-23T14:33:38.278 INFO:tasks.ceph:Generating config... 2024-04-23T14:33:38.279 INFO:tasks.ceph:[osd] osd max object name len = 400 2024-04-23T14:33:38.279 INFO:tasks.ceph:[osd] osd max object namespace len = 64 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-23T14:33:38.280 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-23T14:33:38.280 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-23T14:33:38.280 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-23T14:33:38.280 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-23T14:33:38.280 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-23T14:33:38.280 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-23T14:33:38.280 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-23T14:33:38.281 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-23T14:33:38.281 INFO:tasks.ceph:Setting up mon.a... 2024-04-23T14:33:38.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-23T14:33:38.352 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.keyring 2024-04-23T14:33:38.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-23T14:33:38.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T14:33:38.500 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.84')] 2024-04-23T14:33:38.500 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.84', 'mon election default strategy': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-04-23T14:33:38.501 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:38.501 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T14:33:38.553 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.84 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:38.631 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: generated fsid 75bd7860-6209-40b8-8227-0a2e97ca7f45 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 0 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:fsid 75bd7860-6209-40b8-8227-0a2e97ca7f45 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:last_changed 2024-04-23T14:33:38.632200+0000 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-04-23T14:33:38.632200+0000 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:min_mon_release 17 (quincy) 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:election_strategy: 1 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-23T14:33:38.632 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-23T14:33:38.633 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T14:33:38.682 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 75bd7860-6209-40b8-8227-0a2e97ca7f45... 2024-04-23T14:33:38.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout: chdir = "" 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout: auth supported = cephx 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout: filestore xattr use omap = true 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.753 INFO:teuthology.orchestra.run.smithi084.stdout: mon clock drift allowed = 1.000 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: osd crush chooseleaf type = 0 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: auth debug = true 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on old message = true 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on bug = true 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: mon pg warn max object skew = 0 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: # disable pg_autoscaler by default for new pools 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.754 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default size = 2 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow primary affinity = true 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow pg remap = true 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on legacy crush tunables = false 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on crush straw calc version zero = false 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on no sortbitwise = false 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on osd down out interval zero = false 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on too few osds = false 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: mon_allow_pool_size_one = true 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.755 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: osd default data pool replay window = 5 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: mon allow pool delete = true 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: mon cluster log file level = debug 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: debug asserts on shutdown = true 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: mon health detail to clog = false 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: mon host = 172.21.15.84 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: mon election default strategy = 1 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = 75bd7860-6209-40b8-8227-0a2e97ca7f45 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout:[osd] 2024-04-23T14:33:38.756 INFO:teuthology.orchestra.run.smithi084.stdout: osd journal size = 100 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub load threshold = 5.0 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub max interval = 600 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock profile = high_recovery_ops 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd recover clone overlap = true 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd recovery max chunk = 1048576 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug shutdown = true 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug op order = true 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify stray on activate = true 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug trim objects = true 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: osd open classes on start = true 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug pg log writeout = true 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: osd deep scrub update digest min age = 30 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: osd map max advance = 10 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: journal zero on create = true 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: filestore ondisk finisher threads = 3 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: filestore apply finisher threads = 3 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: bdev debug aio = true 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug misdirected ops = true 2024-04-23T14:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: osd max object name len = 400 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: osd max object namespace len = 64 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore zero block detection = True 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: debug osd = 20 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify cached snaps = True 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify missing on start = True 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock override recovery settings = True 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue = debug_random 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue cut off = debug_random 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.759 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr] 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: debug mgr = 20 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: mgr/telemetry/nag = false 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout:[mon] 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-04-23T14:33:38.760 INFO:teuthology.orchestra.run.smithi084.stdout: debug paxos = 20 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon data avail warn = 5 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon mgr mkfs grace = 240 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd reporter subtree level = osd 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd prime pg temp = true 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: auth mon ticket ttl = 660 # 11m 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: auth service ticket ttl = 240 # 4m 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.761 INFO:teuthology.orchestra.run.smithi084.stdout: # don't complain about insecure global_id in the test suite 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: # 1m isn't quite enough 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon_down_mkfs_grace = 2m 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_filestore_osds = false 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: mon scrub interval = 300 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout:[client] 2024-04-23T14:33:38.762 INFO:teuthology.orchestra.run.smithi084.stdout: rgw cache enabled = true 2024-04-23T14:33:38.763 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable ops log = true 2024-04-23T14:33:38.763 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable usage log = true 2024-04-23T14:33:38.763 INFO:teuthology.orchestra.run.smithi084.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-23T14:33:38.763 INFO:teuthology.orchestra.run.smithi084.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-23T14:33:38.763 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.a] 2024-04-23T14:33:38.764 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-23T14:33:38.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-23T14:33:38.855 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-23T14:33:38.855 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:38.855 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-23T14:33:38.860 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:38.860 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-23T14:33:38.909 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:33:38.909 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:38.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-23T14:33:38.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T14:33:38.974 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:33:38.974 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:39.021 INFO:tasks.ceph:Setting up mon nodes... 2024-04-23T14:33:39.021 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-23T14:33:39.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T14:33:39.106 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T14:33:39.109 INFO:tasks.ceph:Setting up mds nodes... 2024-04-23T14:33:39.109 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-23T14:33:39.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T14:33:39.180 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-23T14:33:39.188 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-23T14:33:39.189 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-04-23T14:33:39.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-23T14:33:39.242 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:39.242 INFO:tasks.ceph:role: osd.0 2024-04-23T14:33:39.242 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:33:39.242 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T14:33:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:33:39.305 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:33:39.752 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-04-23T14:33:39.754 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-04-23T14:33:39.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-23T14:33:39.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-23T14:33:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:33:39.889 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T14:33:39.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-23T14:33:39.942 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:39.942 INFO:tasks.ceph:role: osd.1 2024-04-23T14:33:39.942 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:33:39.942 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-23T14:33:40.005 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:33:40.005 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:33:40.006 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:33:40.467 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-04-23T14:33:40.469 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-04-23T14:33:40.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-23T14:33:40.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-23T14:33:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:33:40.590 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T14:33:40.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-23T14:33:40.642 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:40.642 INFO:tasks.ceph:role: osd.2 2024-04-23T14:33:40.642 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:33:40.642 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-23T14:33:40.705 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:33:40.705 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:33:40.705 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:33:40.706 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T14:33:40.706 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:33:40.706 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:33:40.706 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:33:40.706 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T14:33:40.706 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:33:40.706 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:33:41.177 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-04-23T14:33:41.179 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-04-23T14:33:41.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-23T14:33:41.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-23T14:33:41.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T14:33:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:33:41.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-23T14:33:41.349 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:41.349 INFO:tasks.ceph:role: osd.3 2024-04-23T14:33:41.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:33:41.349 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-23T14:33:41.411 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T14:33:41.412 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T14:33:41.848 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-04-23T14:33:41.851 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-04-23T14:33:41.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-23T14:33:41.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-23T14:33:41.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: /sbin/restorecon: command not found 2024-04-23T14:33:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:33:41.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-23T14:33:42.012 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:42.012 INFO:tasks.ceph:role: osd.4 2024-04-23T14:33:42.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-23T14:33:42.068 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:42.068 INFO:tasks.ceph:role: osd.5 2024-04-23T14:33:42.068 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-23T14:33:42.124 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:42.124 INFO:tasks.ceph:role: osd.6 2024-04-23T14:33:42.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-23T14:33:42.180 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:42.180 INFO:tasks.ceph:role: osd.7 2024-04-23T14:33:42.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-04-23T14:33:42.236 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:42.236 INFO:tasks.ceph:role: osd.8 2024-04-23T14:33:42.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-04-23T14:33:42.292 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T14:33:42.292 INFO:tasks.ceph:role: osd.9 2024-04-23T14:33:42.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:42.410 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:42.408+0000 7fcb7bfa37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-23T14:33:42.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:42.408+0000 7fcb7bfa37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-23T14:33:42.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:42.408+0000 7fcb7bfa37c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T14:33:42.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:42.408+0000 7fcb7bfa37c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T14:33:42.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:42.408+0000 7fcb7bfa37c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T14:33:42.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:42.416+0000 7fcb7bfa37c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-23T14:33:44.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T14:33:44.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:44.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:44.704+0000 7fe023e0b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-23T14:33:44.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:44.704+0000 7fe023e0b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-23T14:33:44.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:44.704+0000 7fe023e0b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T14:33:44.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:44.704+0000 7fe023e0b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T14:33:44.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:44.704+0000 7fe023e0b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T14:33:44.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:44.704+0000 7fe023e0b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-23T14:33:46.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T14:33:46.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:47.010 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:47.008+0000 7f37423e37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-23T14:33:47.010 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:47.008+0000 7f37423e37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-23T14:33:47.010 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:47.008+0000 7f37423e37c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T14:33:47.010 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:47.008+0000 7f37423e37c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T14:33:47.010 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:47.008+0000 7f37423e37c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T14:33:47.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:47.008+0000 7f37423e37c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-23T14:33:49.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T14:33:49.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:49.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:49.332+0000 7fc3e869a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-23T14:33:49.334 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:49.332+0000 7fc3e869a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-23T14:33:49.334 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:49.332+0000 7fc3e869a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T14:33:49.334 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:49.332+0000 7fc3e869a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T14:33:49.334 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:49.332+0000 7fc3e869a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T14:33:49.334 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:49.332+0000 7fc3e869a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-23T14:33:51.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T14:33:51.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:51.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:51.636+0000 7f110de777c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-23T14:33:51.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:51.636+0000 7f110de777c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-23T14:33:51.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:51.636+0000 7f110de777c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T14:33:51.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:51.636+0000 7f110de777c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T14:33:51.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:51.636+0000 7f110de777c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T14:33:51.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:51.636+0000 7f110de777c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-23T14:33:53.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T14:33:53.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:53.950 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:53.948+0000 7fc266dcf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-23T14:33:53.950 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:53.948+0000 7fc266dcf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-23T14:33:53.950 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:53.948+0000 7fc266dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T14:33:53.950 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:53.948+0000 7fc266dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T14:33:53.950 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:53.948+0000 7fc266dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T14:33:53.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:53.948+0000 7fc266dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-23T14:33:56.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T14:33:56.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:56.257 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:56.256+0000 7f8c24b0d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-23T14:33:56.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:56.256+0000 7f8c24b0d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-23T14:33:56.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:56.256+0000 7f8c24b0d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T14:33:56.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:56.256+0000 7f8c24b0d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T14:33:56.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:56.256+0000 7f8c24b0d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T14:33:56.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:56.256+0000 7f8c24b0d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-23T14:33:58.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-23T14:33:58.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:33:58.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:58.584+0000 7fa13ddbd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-23T14:33:58.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:58.584+0000 7fa13ddbd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-23T14:33:58.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:58.584+0000 7fa13ddbd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T14:33:58.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:58.584+0000 7fa13ddbd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T14:33:58.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:58.584+0000 7fa13ddbd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T14:33:58.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:33:58.584+0000 7fa13ddbd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-23T14:34:00.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-23T14:34:00.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:34:00.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:00.892+0000 7f6c3baab7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-04-23T14:34:00.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:00.892+0000 7f6c3baab7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-04-23T14:34:00.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:00.892+0000 7f6c3baab7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-23T14:34:00.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:00.892+0000 7f6c3baab7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-23T14:34:00.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:00.892+0000 7f6c3baab7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-23T14:34:00.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:00.892+0000 7f6c3baab7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-04-23T14:34:03.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-23T14:34:03.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:34:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:03.212+0000 7f54fb3207c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-04-23T14:34:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:03.212+0000 7f54fb3207c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-04-23T14:34:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:03.212+0000 7f54fb3207c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-23T14:34:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:03.212+0000 7f54fb3207c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-23T14:34:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:03.212+0000 7f54fb3207c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-23T14:34:03.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-23T14:34:03.216+0000 7f54fb3207c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-04-23T14:34:05.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-23T14:34:05.457 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-23T14:34:05.457 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-23T14:34:05.511 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-23T14:34:05.564 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-23T14:34:05.619 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-23T14:34:05.672 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-23T14:34:05.728 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-23T14:34:05.784 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-23T14:34:05.840 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-23T14:34:05.895 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-23T14:34:05.948 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:05.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-04-23T14:34:06.004 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:06.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-04-23T14:34:06.060 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:06.060 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-23T14:34:06.109 INFO:tasks.ceph:Adding keys to all mons... 2024-04-23T14:34:06.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-23T14:34:06.162 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.x] 2024-04-23T14:34:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDDxidmNORjBhAAvv2PAp3IZp87Isvtrm6M8w== 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.0] 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDGxidmi7mDGBAAjQgUO6/Hofr2/ewDUrATaA== 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.1] 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDIxidmSsAeKhAAnA4bMgrwtD8xNk5CgKxmAg== 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.2] 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDLxidmeAeiABAAqbARLvq7z9jfSzaTUUdLqw== 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.3] 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDNxidmjA/qExAAiEUzElbr94SAMtK0DZT/Lg== 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.4] 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDPxidmV4wfJhAAECTX6mum1DKMPrPBiQmSow== 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.5] 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDRxidmN8KoOBAAHshP+Wquke0gkxUBobzIbg== 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.6] 2024-04-23T14:34:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDUxidmDLtgDxAAjPnEP5ZVNAw9n5TbcbplDA== 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.7] 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDWxidmvHoOIxAAyrdEiRM39XXRYO+4AKhyHg== 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.8] 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDYxidmaL5TNRAAVIHWikfhaNIbem+hx5jh0w== 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.9] 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDbxidmp8LqDBAACS2S7VTD/ConC/TepMcoOw== 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-04-23T14:34:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDDxidmslHGChAATNKLtNV0/YhTNgD8/6z+Ow== 2024-04-23T14:34:06.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T14:34:06.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:06.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T14:34:07.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T14:34:07.099 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-23T14:34:07.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-23T14:34:07.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T14:34:07.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-23T14:34:07.251 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-23T14:34:07.296 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-23T14:34:07.296 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-23T14:34:07.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-23T14:34:07.341 INFO:tasks.ceph.mon.a:Started 2024-04-23T14:34:07.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph config set mgr mgr_pool false --force 2024-04-23T14:34:07.607 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-23T14:34:07.608 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-23T14:34:07.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-23T14:34:07.610 INFO:tasks.ceph.mgr.x:Started 2024-04-23T14:34:07.610 INFO:tasks.ceph:Setting crush tunables to default 2024-04-23T14:34:07.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd crush tunables default 2024-04-23T14:34:07.809 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-04-23T14:34:07.828 INFO:tasks.ceph:check_enable_crimson: False 2024-04-23T14:34:07.828 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-23T14:34:07.828 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:07.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-23T14:34:07.840 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:07.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-23T14:34:07.895 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:07.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-23T14:34:07.905 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:07.904+0000 7f839982d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:34:07.916 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:07.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-23T14:34:07.971 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:07.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-23T14:34:08.024 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:08.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-23T14:34:08.079 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:08.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-23T14:34:08.130 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:08.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-23T14:34:08.183 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:08.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-04-23T14:34:08.236 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:34:08.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-04-23T14:34:08.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 51365f97-9e49-4fe3-88a1-882cb68532fb 0 2024-04-23T14:34:08.404 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:08.404+0000 7f839982d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:34:08.528 INFO:teuthology.orchestra.run.smithi084.stdout:0 2024-04-23T14:34:08.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new b37a2dd3-7326-40ba-b467-aa87c65d003a 1 2024-04-23T14:34:08.611 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:08.607+0000 7f839982d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:34:08.737 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-04-23T14:34:08.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 28625c48-3473-4b1b-84c4-3958ce03b774 2 2024-04-23T14:34:08.948 INFO:teuthology.orchestra.run.smithi084.stdout:2 2024-04-23T14:34:08.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 8cb99e87-e23c-4955-ac67-d51837a04a22 3 2024-04-23T14:34:09.158 INFO:teuthology.orchestra.run.smithi084.stdout:3 2024-04-23T14:34:09.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 7caefffd-ba0b-4d2a-b5f9-a1c17b6d06f3 4 2024-04-23T14:34:09.177 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.175+0000 7f839982d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:34:09.234 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.231+0000 7f839982d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:34:09.288 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.283+0000 7f839982d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:34:09.366 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.363+0000 7f839982d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:34:09.371 INFO:teuthology.orchestra.run.smithi084.stdout:4 2024-04-23T14:34:09.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 20da4f0c-bdb8-40cb-9a04-823ae3be38eb 5 2024-04-23T14:34:09.421 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.419+0000 7f839982d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:34:09.501 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.499+0000 7f839982d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:34:09.556 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.551+0000 7f839982d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:34:09.582 INFO:teuthology.orchestra.run.smithi084.stdout:5 2024-04-23T14:34:09.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 0ca25a0e-435b-4dd9-b69d-d803cff62eb8 6 2024-04-23T14:34:09.709 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.707+0000 7f839982d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:34:09.773 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:09.771+0000 7f839982d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:34:09.835 INFO:teuthology.orchestra.run.smithi084.stdout:6 2024-04-23T14:34:09.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 63c997f5-c82a-404e-8a1c-35234156c1c7 7 2024-04-23T14:34:10.042 INFO:teuthology.orchestra.run.smithi084.stdout:7 2024-04-23T14:34:10.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new 1a6b6c1c-7f60-4ede-9473-f401bac5c289 8 2024-04-23T14:34:10.276 INFO:teuthology.orchestra.run.smithi084.stdout:8 2024-04-23T14:34:10.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd new c3de37db-80eb-42f7-ad9e-e539d93ec164 9 2024-04-23T14:34:10.367 INFO:tasks.ceph.mgr.x.smithi084.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-23T14:34:10.367 INFO:tasks.ceph.mgr.x.smithi084.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-23T14:34:10.367 INFO:tasks.ceph.mgr.x.smithi084.stderr: from numpy import show_config as show_numpy_config 2024-04-23T14:34:10.430 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:10.427+0000 7f839982d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:34:10.476 INFO:teuthology.orchestra.run.smithi084.stdout:9 2024-04-23T14:34:10.493 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-23T14:34:10.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-23T14:34:10.495 INFO:tasks.ceph.osd.0:Started 2024-04-23T14:34:10.495 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-23T14:34:10.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-23T14:34:10.497 INFO:tasks.ceph.osd.1:Started 2024-04-23T14:34:10.497 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-23T14:34:10.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-23T14:34:10.499 INFO:tasks.ceph.osd.2:Started 2024-04-23T14:34:10.499 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-23T14:34:10.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-23T14:34:10.501 INFO:tasks.ceph.osd.3:Started 2024-04-23T14:34:10.502 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-23T14:34:10.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-23T14:34:10.504 INFO:tasks.ceph.osd.4:Started 2024-04-23T14:34:10.504 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-23T14:34:10.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-23T14:34:10.506 INFO:tasks.ceph.osd.5:Started 2024-04-23T14:34:10.506 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-23T14:34:10.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-23T14:34:10.508 INFO:tasks.ceph.osd.6:Started 2024-04-23T14:34:10.508 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-23T14:34:10.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-23T14:34:10.510 INFO:tasks.ceph.osd.7:Started 2024-04-23T14:34:10.511 INFO:tasks.ceph.osd.8:Restarting daemon 2024-04-23T14:34:10.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-04-23T14:34:10.513 INFO:tasks.ceph.osd.8:Started 2024-04-23T14:34:10.513 INFO:tasks.ceph.osd.9:Restarting daemon 2024-04-23T14:34:10.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-04-23T14:34:10.515 INFO:tasks.ceph.osd.9:Started 2024-04-23T14:34:10.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:34:10.648 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:10.647+0000 7f839982d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:34:10.771 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:34:10.771 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":12,"fsid":"75bd7860-6209-40b8-8227-0a2e97ca7f45","created":"2024-04-23T14:34:07.470391+0000","modified":"2024-04-23T14:34:10.475939+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T14:34:10.475939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51365f97-9e49-4fe3-88a1-882cb68532fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b37a2dd3-7326-40ba-b467-aa87c65d003a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28625c48-3473-4b1b-84c4-3958ce03b774","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8cb99e87-e23c-4955-ac67-d51837a04a22","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7caefffd-ba0b-4d2a-b5f9-a1c17b6d06f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"20da4f0c-bdb8-40cb-9a04-823ae3be38eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0ca25a0e-435b-4dd9-b69d-d803cff62eb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"63c997f5-c82a-404e-8a1c-35234156c1c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"1a6b6c1c-7f60-4ede-9473-f401bac5c289","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"c3de37db-80eb-42f7-ad9e-e539d93ec164","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:34:10.786 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-23T14:34:10.786 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-23T14:34:10.822 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:10.819+0000 7f839982d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:34:10.886 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:10.883+0000 7f839982d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:34:10.913 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:34:10.911+0000 7f51852c57c0 -1 Falling back to public interface 2024-04-23T14:34:10.937 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:34:10.935+0000 7fb56440e7c0 -1 Falling back to public interface 2024-04-23T14:34:10.941 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-04-23T14:34:10.939+0000 7f39b73837c0 -1 Falling back to public interface 2024-04-23T14:34:10.941 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-04-23T14:34:10.939+0000 7fab3774e7c0 -1 Falling back to public interface 2024-04-23T14:34:11.026 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:11.023+0000 7f839982d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:34:11.181 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:34:11.179+0000 7fa95a5137c0 -1 Falling back to public interface 2024-04-23T14:34:11.182 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-04-23T14:34:11.179+0000 7fa9ff7517c0 -1 Falling back to public interface 2024-04-23T14:34:11.189 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-04-23T14:34:11.187+0000 7f6ea61387c0 -1 Falling back to public interface 2024-04-23T14:34:11.189 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:34:11.187+0000 7f3afbb917c0 -1 Falling back to public interface 2024-04-23T14:34:11.201 INFO:tasks.ceph.osd.9.smithi084.stderr:2024-04-23T14:34:11.199+0000 7fa3d7a377c0 -1 Falling back to public interface 2024-04-23T14:34:11.213 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:34:11.211+0000 7f958d2cc7c0 -1 Falling back to public interface 2024-04-23T14:34:11.655 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:11.651+0000 7f839982d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:34:11.729 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:11.727+0000 7f839982d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:34:11.792 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T14:34:11.916 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:11.911+0000 7f839982d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:34:11.967 INFO:teuthology.misc.health.smithi084.stdout: 2024-04-23T14:34:11.967 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":12,"fsid":"75bd7860-6209-40b8-8227-0a2e97ca7f45","created":"2024-04-23T14:34:07.470391+0000","modified":"2024-04-23T14:34:10.475939+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T14:34:10.475939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51365f97-9e49-4fe3-88a1-882cb68532fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b37a2dd3-7326-40ba-b467-aa87c65d003a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28625c48-3473-4b1b-84c4-3958ce03b774","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8cb99e87-e23c-4955-ac67-d51837a04a22","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7caefffd-ba0b-4d2a-b5f9-a1c17b6d06f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"20da4f0c-bdb8-40cb-9a04-823ae3be38eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0ca25a0e-435b-4dd9-b69d-d803cff62eb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"63c997f5-c82a-404e-8a1c-35234156c1c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"1a6b6c1c-7f60-4ede-9473-f401bac5c289","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"c3de37db-80eb-42f7-ad9e-e539d93ec164","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:34:11.981 DEBUG:teuthology.misc:0 of 10 OSDs are up 2024-04-23T14:34:11.991 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:11.987+0000 7f839982d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:34:12.282 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:12.279+0000 7f839982d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:34:12.343 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:12.339+0000 7f839982d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:34:13.320 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-04-23T14:34:13.315+0000 7f39b73837c0 -1 osd.2 0 log_to_monitors true 2024-04-23T14:34:13.371 INFO:tasks.ceph.mgr.x.smithi084.stderr:2024-04-23T14:34:13.367+0000 7f8340c79640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-23T14:34:14.049 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:34:14.047+0000 7f51852c57c0 -1 osd.0 0 log_to_monitors true 2024-04-23T14:34:14.058 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:34:14.055+0000 7fb56440e7c0 -1 osd.5 0 log_to_monitors true 2024-04-23T14:34:14.087 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-04-23T14:34:14.083+0000 7fab3774e7c0 -1 osd.3 0 log_to_monitors true 2024-04-23T14:34:14.189 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-04-23T14:34:14.187+0000 7f6ea61387c0 -1 osd.4 0 log_to_monitors true 2024-04-23T14:34:14.278 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-04-23T14:34:14.275+0000 7fa9ff7517c0 -1 osd.7 0 log_to_monitors true 2024-04-23T14:34:14.290 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:34:14.287+0000 7f958d2cc7c0 -1 osd.8 0 log_to_monitors true 2024-04-23T14:34:14.353 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:34:14.351+0000 7f3afbb917c0 -1 osd.6 0 log_to_monitors true 2024-04-23T14:34:14.368 INFO:tasks.ceph.osd.9.smithi084.stderr:2024-04-23T14:34:14.367+0000 7fa3d7a377c0 -1 osd.9 0 log_to_monitors true 2024-04-23T14:34:14.371 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:34:14.367+0000 7fa95a5137c0 -1 osd.1 0 log_to_monitors true 2024-04-23T14:34:15.123 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-04-23T14:34:15.119+0000 7f39b332c640 -1 osd.2 0 waiting for initial osdmap 2024-04-23T14:34:15.137 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-04-23T14:34:15.135+0000 7f39ab160640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:15.580 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:34:15.575+0000 7f518126e640 -1 osd.0 0 waiting for initial osdmap 2024-04-23T14:34:15.580 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-04-23T14:34:15.575+0000 7f6ea20e1640 -1 osd.4 0 waiting for initial osdmap 2024-04-23T14:34:15.580 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:34:15.579+0000 7fb5603b7640 -1 osd.5 0 waiting for initial osdmap 2024-04-23T14:34:15.581 INFO:tasks.ceph.osd.9.smithi084.stderr:2024-04-23T14:34:15.579+0000 7fa3d39e0640 -1 osd.9 0 waiting for initial osdmap 2024-04-23T14:34:15.581 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:34:15.579+0000 7f9589275640 -1 osd.8 0 waiting for initial osdmap 2024-04-23T14:34:15.698 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:34:15.695+0000 7f517d0aa640 -1 osd.0 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:16.099 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:34:16.095+0000 7f95850b1640 -1 osd.8 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:16.290 INFO:tasks.ceph.osd.9.smithi084.stderr:2024-04-23T14:34:16.287+0000 7fa3cf81c640 -1 osd.9 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:16.291 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:34:16.287+0000 7fb55c1f3640 -1 osd.5 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:16.396 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-04-23T14:34:16.395+0000 7f6e9df1d640 -1 osd.4 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:16.930 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:34:16.927+0000 7fa9564bc640 -1 osd.1 0 waiting for initial osdmap 2024-04-23T14:34:16.970 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:34:16.967+0000 7fa94e2f0640 -1 osd.1 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:16.995 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-04-23T14:34:16.991+0000 7fab336f7640 -1 osd.3 0 waiting for initial osdmap 2024-04-23T14:34:17.032 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-04-23T14:34:17.027+0000 7fab2b52b640 -1 osd.3 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:17.402 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-04-23T14:34:17.399+0000 7fa9fb6fa640 -1 osd.7 0 waiting for initial osdmap 2024-04-23T14:34:17.423 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:34:17.419+0000 7f3af7b3a640 -1 osd.6 0 waiting for initial osdmap 2024-04-23T14:34:17.426 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-04-23T14:34:17.423+0000 7fa9f352e640 -1 osd.7 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:17.449 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:34:17.447+0000 7f3aef96e640 -1 osd.6 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:34:18.987 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T14:34:19.289 INFO:teuthology.misc.health.smithi084.stdout: 2024-04-23T14:34:19.289 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":18,"fsid":"75bd7860-6209-40b8-8227-0a2e97ca7f45","created":"2024-04-23T14:34:07.470391+0000","modified":"2024-04-23T14:34:18.999493+0000","last_up_change":"2024-04-23T14:34:17.997668+0000","last_in_change":"2024-04-23T14:34:10.475939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51365f97-9e49-4fe3-88a1-882cb68532fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6801","nonce":1294248214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6803","nonce":1294248214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6807","nonce":1294248214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6805","nonce":1294248214}]},"public_addr":"172.21.15.84:6801/1294248214","cluster_addr":"172.21.15.84:6803/1294248214","heartbeat_back_addr":"172.21.15.84:6807/1294248214","heartbeat_front_addr":"172.21.15.84:6805/1294248214","state":["exists","up"]},{"osd":1,"uuid":"b37a2dd3-7326-40ba-b467-aa87c65d003a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6834","nonce":4126848561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6838","nonce":4126848561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6846","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6851","nonce":4126848561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6842","nonce":4126848561}]},"public_addr":"172.21.15.84:6834/4126848561","cluster_addr":"172.21.15.84:6838/4126848561","heartbeat_back_addr":"172.21.15.84:6851/4126848561","heartbeat_front_addr":"172.21.15.84:6842/4126848561","state":["exists","up"]},{"osd":2,"uuid":"28625c48-3473-4b1b-84c4-3958ce03b774","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6818","nonce":187511398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6822","nonce":187511398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6830","nonce":187511398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6826","nonce":187511398}]},"public_addr":"172.21.15.84:6818/187511398","cluster_addr":"172.21.15.84:6822/187511398","heartbeat_back_addr":"172.21.15.84:6830/187511398","heartbeat_front_addr":"172.21.15.84:6826/187511398","state":["exists","up"]},{"osd":3,"uuid":"8cb99e87-e23c-4955-ac67-d51837a04a22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6817","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6819","nonce":2048464647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6821","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6823","nonce":2048464647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6829","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6831","nonce":2048464647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6825","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6827","nonce":2048464647}]},"public_addr":"172.21.15.84:6819/2048464647","cluster_addr":"172.21.15.84:6823/2048464647","heartbeat_back_addr":"172.21.15.84:6831/2048464647","heartbeat_front_addr":"172.21.15.84:6827/2048464647","state":["exists","up"]},{"osd":4,"uuid":"7caefffd-ba0b-4d2a-b5f9-a1c17b6d06f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6843","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6848","nonce":575087180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6852","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6854","nonce":575087180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6858","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6860","nonce":575087180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6855","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6857","nonce":575087180}]},"public_addr":"172.21.15.84:6848/575087180","cluster_addr":"172.21.15.84:6854/575087180","heartbeat_back_addr":"172.21.15.84:6860/575087180","heartbeat_front_addr":"172.21.15.84:6857/575087180","state":["exists","up"]},{"osd":5,"uuid":"20da4f0c-bdb8-40cb-9a04-823ae3be38eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6809","nonce":1453654031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6811","nonce":1453654031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6815","nonce":1453654031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6813","nonce":1453654031}]},"public_addr":"172.21.15.84:6809/1453654031","cluster_addr":"172.21.15.84:6811/1453654031","heartbeat_back_addr":"172.21.15.84:6815/1453654031","heartbeat_front_addr":"172.21.15.84:6813/1453654031","state":["exists","up"]},{"osd":6,"uuid":"0ca25a0e-435b-4dd9-b69d-d803cff62eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6844","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6849","nonce":832042711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6853","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6856","nonce":832042711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6864","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6866","nonce":832042711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6859","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6862","nonce":832042711}]},"public_addr":"172.21.15.84:6849/832042711","cluster_addr":"172.21.15.84:6856/832042711","heartbeat_back_addr":"172.21.15.84:6866/832042711","heartbeat_front_addr":"172.21.15.84:6862/832042711","state":["exists","up"]},{"osd":7,"uuid":"63c997f5-c82a-404e-8a1c-35234156c1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6833","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6835","nonce":3079998612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6837","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6839","nonce":3079998612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6847","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6850","nonce":3079998612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6841","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6845","nonce":3079998612}]},"public_addr":"172.21.15.84:6835/3079998612","cluster_addr":"172.21.15.84:6839/3079998612","heartbeat_back_addr":"172.21.15.84:6850/3079998612","heartbeat_front_addr":"172.21.15.84:6845/3079998612","state":["exists","up"]},{"osd":8,"uuid":"1a6b6c1c-7f60-4ede-9473-f401bac5c289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6872","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6873","nonce":1867330816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6874","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6875","nonce":1867330816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6878","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6879","nonce":1867330816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6876","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6877","nonce":1867330816}]},"public_addr":"172.21.15.84:6873/1867330816","cluster_addr":"172.21.15.84:6875/1867330816","heartbeat_back_addr":"172.21.15.84:6879/1867330816","heartbeat_front_addr":"172.21.15.84:6877/1867330816","state":["exists","up"]},{"osd":9,"uuid":"c3de37db-80eb-42f7-ad9e-e539d93ec164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6861","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6863","nonce":3429805588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6865","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6867","nonce":3429805588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6870","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6871","nonce":3429805588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6868","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6869","nonce":3429805588}]},"public_addr":"172.21.15.84:6863/3429805588","cluster_addr":"172.21.15.84:6867/3429805588","heartbeat_back_addr":"172.21.15.84:6871/3429805588","heartbeat_front_addr":"172.21.15.84:6869/3429805588","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.009829+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.325729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:14.359911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.126904+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.150634+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.070198+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.375482+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.280610+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.331276+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.372023+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:34:19.306 DEBUG:teuthology.misc:10 of 10 OSDs are up 2024-04-23T14:34:19.306 INFO:tasks.ceph:Creating RBD pool 2024-04-23T14:34:19.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-23T14:34:20.004 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'rbd' created 2024-04-23T14:34:20.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-23T14:34:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-23T14:34:21.041 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-23T14:34:21.041 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T14:34:21.041 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-23T14:34:21.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-23T14:34:21.042 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-23T14:34:21.388 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:34:21.406 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6880","nonce":4215258470},{"type":"v1","addr":"172.21.15.84:6881","nonce":4215258470}]},"active_addr":"172.21.15.84:6881/4215258470","active_change":"2024-04-23T14:34:12.363068+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":2782755380}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":743812773}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":403629785}]}]} 2024-04-23T14:34:21.408 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-23T14:34:21.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-23T14:34:21.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:34:21.697 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:34:21.697 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":20,"fsid":"75bd7860-6209-40b8-8227-0a2e97ca7f45","created":"2024-04-23T14:34:07.470391+0000","modified":"2024-04-23T14:34:21.010829+0000","last_up_change":"2024-04-23T14:34:17.997668+0000","last_in_change":"2024-04-23T14:34:10.475939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T14:34:19.601403+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51365f97-9e49-4fe3-88a1-882cb68532fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6801","nonce":1294248214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6803","nonce":1294248214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6807","nonce":1294248214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6805","nonce":1294248214}]},"public_addr":"172.21.15.84:6801/1294248214","cluster_addr":"172.21.15.84:6803/1294248214","heartbeat_back_addr":"172.21.15.84:6807/1294248214","heartbeat_front_addr":"172.21.15.84:6805/1294248214","state":["exists","up"]},{"osd":1,"uuid":"b37a2dd3-7326-40ba-b467-aa87c65d003a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6834","nonce":4126848561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6838","nonce":4126848561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6846","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6851","nonce":4126848561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6842","nonce":4126848561}]},"public_addr":"172.21.15.84:6834/4126848561","cluster_addr":"172.21.15.84:6838/4126848561","heartbeat_back_addr":"172.21.15.84:6851/4126848561","heartbeat_front_addr":"172.21.15.84:6842/4126848561","state":["exists","up"]},{"osd":2,"uuid":"28625c48-3473-4b1b-84c4-3958ce03b774","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6818","nonce":187511398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6822","nonce":187511398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6830","nonce":187511398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6826","nonce":187511398}]},"public_addr":"172.21.15.84:6818/187511398","cluster_addr":"172.21.15.84:6822/187511398","heartbeat_back_addr":"172.21.15.84:6830/187511398","heartbeat_front_addr":"172.21.15.84:6826/187511398","state":["exists","up"]},{"osd":3,"uuid":"8cb99e87-e23c-4955-ac67-d51837a04a22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6817","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6819","nonce":2048464647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6821","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6823","nonce":2048464647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6829","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6831","nonce":2048464647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6825","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6827","nonce":2048464647}]},"public_addr":"172.21.15.84:6819/2048464647","cluster_addr":"172.21.15.84:6823/2048464647","heartbeat_back_addr":"172.21.15.84:6831/2048464647","heartbeat_front_addr":"172.21.15.84:6827/2048464647","state":["exists","up"]},{"osd":4,"uuid":"7caefffd-ba0b-4d2a-b5f9-a1c17b6d06f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6843","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6848","nonce":575087180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6852","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6854","nonce":575087180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6858","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6860","nonce":575087180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6855","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6857","nonce":575087180}]},"public_addr":"172.21.15.84:6848/575087180","cluster_addr":"172.21.15.84:6854/575087180","heartbeat_back_addr":"172.21.15.84:6860/575087180","heartbeat_front_addr":"172.21.15.84:6857/575087180","state":["exists","up"]},{"osd":5,"uuid":"20da4f0c-bdb8-40cb-9a04-823ae3be38eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6809","nonce":1453654031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6811","nonce":1453654031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6815","nonce":1453654031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6813","nonce":1453654031}]},"public_addr":"172.21.15.84:6809/1453654031","cluster_addr":"172.21.15.84:6811/1453654031","heartbeat_back_addr":"172.21.15.84:6815/1453654031","heartbeat_front_addr":"172.21.15.84:6813/1453654031","state":["exists","up"]},{"osd":6,"uuid":"0ca25a0e-435b-4dd9-b69d-d803cff62eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6844","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6849","nonce":832042711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6853","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6856","nonce":832042711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6864","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6866","nonce":832042711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6859","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6862","nonce":832042711}]},"public_addr":"172.21.15.84:6849/832042711","cluster_addr":"172.21.15.84:6856/832042711","heartbeat_back_addr":"172.21.15.84:6866/832042711","heartbeat_front_addr":"172.21.15.84:6862/832042711","state":["exists","up"]},{"osd":7,"uuid":"63c997f5-c82a-404e-8a1c-35234156c1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6833","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6835","nonce":3079998612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6837","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6839","nonce":3079998612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6847","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6850","nonce":3079998612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6841","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6845","nonce":3079998612}]},"public_addr":"172.21.15.84:6835/3079998612","cluster_addr":"172.21.15.84:6839/3079998612","heartbeat_back_addr":"172.21.15.84:6850/3079998612","heartbeat_front_addr":"172.21.15.84:6845/3079998612","state":["exists","up"]},{"osd":8,"uuid":"1a6b6c1c-7f60-4ede-9473-f401bac5c289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6872","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6873","nonce":1867330816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6874","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6875","nonce":1867330816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6878","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6879","nonce":1867330816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6876","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6877","nonce":1867330816}]},"public_addr":"172.21.15.84:6873/1867330816","cluster_addr":"172.21.15.84:6875/1867330816","heartbeat_back_addr":"172.21.15.84:6879/1867330816","heartbeat_front_addr":"172.21.15.84:6877/1867330816","state":["exists","up"]},{"osd":9,"uuid":"c3de37db-80eb-42f7-ad9e-e539d93ec164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6861","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6863","nonce":3429805588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6865","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6867","nonce":3429805588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6870","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6871","nonce":3429805588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6868","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6869","nonce":3429805588}]},"public_addr":"172.21.15.84:6863/3429805588","cluster_addr":"172.21.15.84:6867/3429805588","heartbeat_back_addr":"172.21.15.84:6871/3429805588","heartbeat_front_addr":"172.21.15.84:6869/3429805588","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.009829+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.325729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:14.359911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.126904+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.150634+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.070198+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.375482+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.280610+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.331276+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.372023+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:34:21.714 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-23T14:34:21.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:34:21.994 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:34:21.994 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":20,"fsid":"75bd7860-6209-40b8-8227-0a2e97ca7f45","created":"2024-04-23T14:34:07.470391+0000","modified":"2024-04-23T14:34:21.010829+0000","last_up_change":"2024-04-23T14:34:17.997668+0000","last_in_change":"2024-04-23T14:34:10.475939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T14:34:19.601403+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51365f97-9e49-4fe3-88a1-882cb68532fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6801","nonce":1294248214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6803","nonce":1294248214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6807","nonce":1294248214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6805","nonce":1294248214}]},"public_addr":"172.21.15.84:6801/1294248214","cluster_addr":"172.21.15.84:6803/1294248214","heartbeat_back_addr":"172.21.15.84:6807/1294248214","heartbeat_front_addr":"172.21.15.84:6805/1294248214","state":["exists","up"]},{"osd":1,"uuid":"b37a2dd3-7326-40ba-b467-aa87c65d003a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6834","nonce":4126848561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6838","nonce":4126848561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6846","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6851","nonce":4126848561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6842","nonce":4126848561}]},"public_addr":"172.21.15.84:6834/4126848561","cluster_addr":"172.21.15.84:6838/4126848561","heartbeat_back_addr":"172.21.15.84:6851/4126848561","heartbeat_front_addr":"172.21.15.84:6842/4126848561","state":["exists","up"]},{"osd":2,"uuid":"28625c48-3473-4b1b-84c4-3958ce03b774","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6818","nonce":187511398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6822","nonce":187511398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6830","nonce":187511398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6826","nonce":187511398}]},"public_addr":"172.21.15.84:6818/187511398","cluster_addr":"172.21.15.84:6822/187511398","heartbeat_back_addr":"172.21.15.84:6830/187511398","heartbeat_front_addr":"172.21.15.84:6826/187511398","state":["exists","up"]},{"osd":3,"uuid":"8cb99e87-e23c-4955-ac67-d51837a04a22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6817","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6819","nonce":2048464647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6821","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6823","nonce":2048464647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6829","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6831","nonce":2048464647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6825","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6827","nonce":2048464647}]},"public_addr":"172.21.15.84:6819/2048464647","cluster_addr":"172.21.15.84:6823/2048464647","heartbeat_back_addr":"172.21.15.84:6831/2048464647","heartbeat_front_addr":"172.21.15.84:6827/2048464647","state":["exists","up"]},{"osd":4,"uuid":"7caefffd-ba0b-4d2a-b5f9-a1c17b6d06f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6843","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6848","nonce":575087180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6852","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6854","nonce":575087180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6858","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6860","nonce":575087180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6855","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6857","nonce":575087180}]},"public_addr":"172.21.15.84:6848/575087180","cluster_addr":"172.21.15.84:6854/575087180","heartbeat_back_addr":"172.21.15.84:6860/575087180","heartbeat_front_addr":"172.21.15.84:6857/575087180","state":["exists","up"]},{"osd":5,"uuid":"20da4f0c-bdb8-40cb-9a04-823ae3be38eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6809","nonce":1453654031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6811","nonce":1453654031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6815","nonce":1453654031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6813","nonce":1453654031}]},"public_addr":"172.21.15.84:6809/1453654031","cluster_addr":"172.21.15.84:6811/1453654031","heartbeat_back_addr":"172.21.15.84:6815/1453654031","heartbeat_front_addr":"172.21.15.84:6813/1453654031","state":["exists","up"]},{"osd":6,"uuid":"0ca25a0e-435b-4dd9-b69d-d803cff62eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6844","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6849","nonce":832042711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6853","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6856","nonce":832042711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6864","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6866","nonce":832042711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6859","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6862","nonce":832042711}]},"public_addr":"172.21.15.84:6849/832042711","cluster_addr":"172.21.15.84:6856/832042711","heartbeat_back_addr":"172.21.15.84:6866/832042711","heartbeat_front_addr":"172.21.15.84:6862/832042711","state":["exists","up"]},{"osd":7,"uuid":"63c997f5-c82a-404e-8a1c-35234156c1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6833","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6835","nonce":3079998612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6837","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6839","nonce":3079998612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6847","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6850","nonce":3079998612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6841","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6845","nonce":3079998612}]},"public_addr":"172.21.15.84:6835/3079998612","cluster_addr":"172.21.15.84:6839/3079998612","heartbeat_back_addr":"172.21.15.84:6850/3079998612","heartbeat_front_addr":"172.21.15.84:6845/3079998612","state":["exists","up"]},{"osd":8,"uuid":"1a6b6c1c-7f60-4ede-9473-f401bac5c289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6872","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6873","nonce":1867330816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6874","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6875","nonce":1867330816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6878","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6879","nonce":1867330816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6876","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6877","nonce":1867330816}]},"public_addr":"172.21.15.84:6873/1867330816","cluster_addr":"172.21.15.84:6875/1867330816","heartbeat_back_addr":"172.21.15.84:6879/1867330816","heartbeat_front_addr":"172.21.15.84:6877/1867330816","state":["exists","up"]},{"osd":9,"uuid":"c3de37db-80eb-42f7-ad9e-e539d93ec164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6861","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6863","nonce":3429805588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6865","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6867","nonce":3429805588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6870","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6871","nonce":3429805588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6868","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6869","nonce":3429805588}]},"public_addr":"172.21.15.84:6863/3429805588","cluster_addr":"172.21.15.84:6867/3429805588","heartbeat_back_addr":"172.21.15.84:6871/3429805588","heartbeat_front_addr":"172.21.15.84:6869/3429805588","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.009829+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.325729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:14.359911+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.126904+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.150634+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.070198+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.375482+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.280610+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.331276+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.372023+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:34:22.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-23T14:34:22.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-23T14:34:22.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-23T14:34:22.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-23T14:34:22.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-23T14:34:22.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-23T14:34:22.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-23T14:34:22.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-23T14:34:22.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-23T14:34:22.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-23T14:34:22.280 INFO:teuthology.orchestra.run.smithi084.stdout:64424509444 2024-04-23T14:34:22.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T14:34:22.303 INFO:teuthology.orchestra.run.smithi084.stdout:68719476739 2024-04-23T14:34:22.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-23T14:34:22.305 INFO:teuthology.orchestra.run.smithi084.stdout:68719476739 2024-04-23T14:34:22.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T14:34:22.308 INFO:teuthology.orchestra.run.smithi084.stdout:68719476738 2024-04-23T14:34:22.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T14:34:22.326 INFO:teuthology.orchestra.run.smithi084.stdout:68719476739 2024-04-23T14:34:22.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T14:34:22.337 INFO:teuthology.orchestra.run.smithi084.stdout:73014444035 2024-04-23T14:34:22.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T14:34:22.349 INFO:teuthology.orchestra.run.smithi084.stdout:73014444035 2024-04-23T14:34:22.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T14:34:22.356 INFO:teuthology.orchestra.run.smithi084.stdout:73014444034 2024-04-23T14:34:22.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T14:34:22.384 INFO:teuthology.orchestra.run.smithi084.stdout:68719476740 2024-04-23T14:34:22.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-23T14:34:22.408 INFO:teuthology.orchestra.run.smithi084.stdout:73014444035 2024-04-23T14:34:22.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T14:34:22.865 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-04-23T14:34:22.921 INFO:teuthology.orchestra.run.smithi084.stdout:68719476737 2024-04-23T14:34:22.922 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 1 for osd.9 2024-04-23T14:34:22.943 INFO:teuthology.orchestra.run.smithi084.stdout:68719476738 2024-04-23T14:34:22.948 INFO:teuthology.orchestra.run.smithi084.stdout:73014444034 2024-04-23T14:34:22.960 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476738 got 68719476737 for osd.5 2024-04-23T14:34:22.970 INFO:teuthology.orchestra.run.smithi084.stdout:64424509442 2024-04-23T14:34:22.971 INFO:teuthology.orchestra.run.smithi084.stdout:68719476738 2024-04-23T14:34:22.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444034 for osd.7 2024-04-23T14:34:22.983 INFO:teuthology.orchestra.run.smithi084.stdout:73014444034 2024-04-23T14:34:22.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476738 for osd.4 2024-04-23T14:34:23.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509442 for osd.2 2024-04-23T14:34:23.002 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476738 for osd.0 2024-04-23T14:34:23.002 INFO:teuthology.orchestra.run.smithi084.stdout:73014444033 2024-04-23T14:34:23.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444034 for osd.1 2024-04-23T14:34:23.026 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444034 got 73014444033 for osd.3 2024-04-23T14:34:23.069 INFO:teuthology.orchestra.run.smithi084.stdout:73014444034 2024-04-23T14:34:23.085 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444034 for osd.6 2024-04-23T14:34:23.095 INFO:teuthology.orchestra.run.smithi084.stdout:68719476740 2024-04-23T14:34:23.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476740 for osd.8 2024-04-23T14:34:23.111 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:23.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-23T14:34:23.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T14:34:23.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T14:34:23.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T14:34:24.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T14:34:24.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T14:34:24.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T14:34:24.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T14:34:24.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T14:34:24.522 INFO:teuthology.orchestra.run.smithi084.stdout:68719476738 2024-04-23T14:34:24.525 INFO:teuthology.orchestra.run.smithi084.stdout:68719476739 2024-04-23T14:34:24.551 INFO:teuthology.orchestra.run.smithi084.stdout:64424509444 2024-04-23T14:34:24.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476738 got 68719476738 for osd.5 2024-04-23T14:34:24.552 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.559 INFO:teuthology.orchestra.run.smithi084.stdout:68719476739 2024-04-23T14:34:24.562 INFO:teuthology.orchestra.run.smithi084.stdout:68719476739 2024-04-23T14:34:24.570 INFO:teuthology.orchestra.run.smithi084.stdout:73014444035 2024-04-23T14:34:24.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.9 2024-04-23T14:34:24.576 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.592 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.2 2024-04-23T14:34:24.592 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.4 2024-04-23T14:34:24.597 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476739 for osd.0 2024-04-23T14:34:24.598 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.601 INFO:teuthology.orchestra.run.smithi084.stdout:73014444035 2024-04-23T14:34:24.603 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444035 for osd.1 2024-04-23T14:34:24.604 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.610 INFO:teuthology.orchestra.run.smithi084.stdout:73014444034 2024-04-23T14:34:24.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444035 for osd.7 2024-04-23T14:34:24.620 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444034 got 73014444034 for osd.3 2024-04-23T14:34:24.628 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:24.661 INFO:teuthology.orchestra.run.smithi084.stdout:73014444034 2024-04-23T14:34:24.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444034 for osd.6 2024-04-23T14:34:25.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T14:34:25.971 INFO:teuthology.orchestra.run.smithi084.stdout:73014444036 2024-04-23T14:34:25.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444035 got 73014444036 for osd.6 2024-04-23T14:34:25.988 DEBUG:teuthology.parallel:result is None 2024-04-23T14:34:25.988 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T14:34:25.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:34:26.266 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:34:26.266 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:34:26.282 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-23T14:34:24.373431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":272032,"kb_used_data":1808,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048303968,"statfs":{"total":1073741824000,"available":1073463263232,"internally_reserved":0,"allocated":1851392,"data_stored":589040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.027884+0000","last_change":"2024-04-23T14:34:21.100972+0000","last_active":"2024-04-23T14:34:22.027884+0000","last_peered":"2024-04-23T14:34:22.027884+0000","last_clean":"2024-04-23T14:34:22.027884+0000","last_became_active":"2024-04-23T14:34:21.100585+0000","last_became_peered":"2024-04-23T14:34:21.100585+0000","last_unstale":"2024-04-23T14:34:22.027884+0000","last_undegraded":"2024-04-23T14:34:22.027884+0000","last_fullsized":"2024-04-23T14:34:22.027884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:30.807460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.027661+0000","last_change":"2024-04-23T14:34:21.411185+0000","last_active":"2024-04-23T14:34:22.027661+0000","last_peered":"2024-04-23T14:34:22.027661+0000","last_clean":"2024-04-23T14:34:22.027661+0000","last_became_active":"2024-04-23T14:34:21.410862+0000","last_became_peered":"2024-04-23T14:34:21.410862+0000","last_unstale":"2024-04-23T14:34:22.027661+0000","last_undegraded":"2024-04-23T14:34:22.027661+0000","last_fullsized":"2024-04-23T14:34:22.027661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:49:12.795767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.580461+0000","last_change":"2024-04-23T14:34:21.027159+0000","last_active":"2024-04-23T14:34:22.580461+0000","last_peered":"2024-04-23T14:34:22.580461+0000","last_clean":"2024-04-23T14:34:22.580461+0000","last_became_active":"2024-04-23T14:34:21.026785+0000","last_became_peered":"2024-04-23T14:34:21.026785+0000","last_unstale":"2024-04-23T14:34:22.580461+0000","last_undegraded":"2024-04-23T14:34:22.580461+0000","last_fullsized":"2024-04-23T14:34:22.580461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:15:28.076531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.580606+0000","last_change":"2024-04-23T14:34:21.026562+0000","last_active":"2024-04-23T14:34:22.580606+0000","last_peered":"2024-04-23T14:34:22.580606+0000","last_clean":"2024-04-23T14:34:22.580606+0000","last_became_active":"2024-04-23T14:34:21.026171+0000","last_became_peered":"2024-04-23T14:34:21.026171+0000","last_unstale":"2024-04-23T14:34:22.580606+0000","last_undegraded":"2024-04-23T14:34:22.580606+0000","last_fullsized":"2024-04-23T14:34:22.580606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:49:12.795767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.580303+0000","last_change":"2024-04-23T14:34:21.256567+0000","last_active":"2024-04-23T14:34:22.580303+0000","last_peered":"2024-04-23T14:34:22.580303+0000","last_clean":"2024-04-23T14:34:22.580303+0000","last_became_active":"2024-04-23T14:34:21.255813+0000","last_became_peered":"2024-04-23T14:34:21.255813+0000","last_unstale":"2024-04-23T14:34:22.580303+0000","last_undegraded":"2024-04-23T14:34:22.580303+0000","last_fullsized":"2024-04-23T14:34:22.580303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:39.627294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.575759+0000","last_change":"2024-04-23T14:34:21.100831+0000","last_active":"2024-04-23T14:34:22.575759+0000","last_peered":"2024-04-23T14:34:22.575759+0000","last_clean":"2024-04-23T14:34:22.575759+0000","last_became_active":"2024-04-23T14:34:21.100554+0000","last_became_peered":"2024-04-23T14:34:21.100554+0000","last_unstale":"2024-04-23T14:34:22.575759+0000","last_undegraded":"2024-04-23T14:34:22.575759+0000","last_fullsized":"2024-04-23T14:34:22.575759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:18.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-23T14:34:21.025987+0000","last_change":"2024-04-23T14:34:21.025417+0000","last_active":"2024-04-23T14:34:21.025987+0000","last_peered":"2024-04-23T14:34:21.025987+0000","last_clean":"2024-04-23T14:34:21.025987+0000","last_became_active":"2024-04-23T14:34:21.024760+0000","last_became_peered":"2024-04-23T14:34:21.024760+0000","last_unstale":"2024-04-23T14:34:21.025987+0000","last_undegraded":"2024-04-23T14:34:21.025987+0000","last_fullsized":"2024-04-23T14:34:21.025987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:16.223192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-23T14:34:21.027774+0000","last_change":"2024-04-23T14:34:21.026063+0000","last_active":"2024-04-23T14:34:21.027774+0000","last_peered":"2024-04-23T14:34:21.027774+0000","last_clean":"2024-04-23T14:34:21.027774+0000","last_became_active":"2024-04-23T14:34:21.025781+0000","last_became_peered":"2024-04-23T14:34:21.025781+0000","last_unstale":"2024-04-23T14:34:21.027774+0000","last_undegraded":"2024-04-23T14:34:21.027774+0000","last_fullsized":"2024-04-23T14:34:21.027774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:16.223192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830384,"statfs":{"total":107374182400,"available":107346313216,"internally_reserved":0,"allocated":200704,"data_stored":66933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830376,"statfs":{"total":107374182400,"available":107346305024,"internally_reserved":0,"allocated":200704,"data_stored":66933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":17,"seq":73014444035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830384,"statfs":{"total":107374182400,"available":107346313216,"internally_reserved":0,"allocated":200704,"data_stored":66933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27192,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830408,"statfs":{"total":107374182400,"available":107346337792,"internally_reserved":0,"allocated":176128,"data_stored":54653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,4,5,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830396,"statfs":{"total":107374182400,"available":107346325504,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,5,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":29503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:34:26.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:34:26.550 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:34:26.551 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:34:26.567 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-23T14:34:26.374167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":272032,"kb_used_data":1808,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048303968,"statfs":{"total":1073741824000,"available":1073463263232,"internally_reserved":0,"allocated":1851392,"data_stored":589040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.027884+0000","last_change":"2024-04-23T14:34:21.100972+0000","last_active":"2024-04-23T14:34:22.027884+0000","last_peered":"2024-04-23T14:34:22.027884+0000","last_clean":"2024-04-23T14:34:22.027884+0000","last_became_active":"2024-04-23T14:34:21.100585+0000","last_became_peered":"2024-04-23T14:34:21.100585+0000","last_unstale":"2024-04-23T14:34:22.027884+0000","last_undegraded":"2024-04-23T14:34:22.027884+0000","last_fullsized":"2024-04-23T14:34:22.027884+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:20:30.807460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.027661+0000","last_change":"2024-04-23T14:34:21.411185+0000","last_active":"2024-04-23T14:34:22.027661+0000","last_peered":"2024-04-23T14:34:22.027661+0000","last_clean":"2024-04-23T14:34:22.027661+0000","last_became_active":"2024-04-23T14:34:21.410862+0000","last_became_peered":"2024-04-23T14:34:21.410862+0000","last_unstale":"2024-04-23T14:34:22.027661+0000","last_undegraded":"2024-04-23T14:34:22.027661+0000","last_fullsized":"2024-04-23T14:34:22.027661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:49:12.795767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.580461+0000","last_change":"2024-04-23T14:34:21.027159+0000","last_active":"2024-04-23T14:34:22.580461+0000","last_peered":"2024-04-23T14:34:22.580461+0000","last_clean":"2024-04-23T14:34:22.580461+0000","last_became_active":"2024-04-23T14:34:21.026785+0000","last_became_peered":"2024-04-23T14:34:21.026785+0000","last_unstale":"2024-04-23T14:34:22.580461+0000","last_undegraded":"2024-04-23T14:34:22.580461+0000","last_fullsized":"2024-04-23T14:34:22.580461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:15:28.076531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.580606+0000","last_change":"2024-04-23T14:34:21.026562+0000","last_active":"2024-04-23T14:34:22.580606+0000","last_peered":"2024-04-23T14:34:22.580606+0000","last_clean":"2024-04-23T14:34:22.580606+0000","last_became_active":"2024-04-23T14:34:21.026171+0000","last_became_peered":"2024-04-23T14:34:21.026171+0000","last_unstale":"2024-04-23T14:34:22.580606+0000","last_undegraded":"2024-04-23T14:34:22.580606+0000","last_fullsized":"2024-04-23T14:34:22.580606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:49:12.795767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.580303+0000","last_change":"2024-04-23T14:34:21.256567+0000","last_active":"2024-04-23T14:34:22.580303+0000","last_peered":"2024-04-23T14:34:22.580303+0000","last_clean":"2024-04-23T14:34:22.580303+0000","last_became_active":"2024-04-23T14:34:21.255813+0000","last_became_peered":"2024-04-23T14:34:21.255813+0000","last_unstale":"2024-04-23T14:34:22.580303+0000","last_undegraded":"2024-04-23T14:34:22.580303+0000","last_fullsized":"2024-04-23T14:34:22.580303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:39.627294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-23T14:34:22.575759+0000","last_change":"2024-04-23T14:34:21.100831+0000","last_active":"2024-04-23T14:34:22.575759+0000","last_peered":"2024-04-23T14:34:22.575759+0000","last_clean":"2024-04-23T14:34:22.575759+0000","last_became_active":"2024-04-23T14:34:21.100554+0000","last_became_peered":"2024-04-23T14:34:21.100554+0000","last_unstale":"2024-04-23T14:34:22.575759+0000","last_undegraded":"2024-04-23T14:34:22.575759+0000","last_fullsized":"2024-04-23T14:34:22.575759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:00:18.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-23T14:34:21.025987+0000","last_change":"2024-04-23T14:34:21.025417+0000","last_active":"2024-04-23T14:34:21.025987+0000","last_peered":"2024-04-23T14:34:21.025987+0000","last_clean":"2024-04-23T14:34:21.025987+0000","last_became_active":"2024-04-23T14:34:21.024760+0000","last_became_peered":"2024-04-23T14:34:21.024760+0000","last_unstale":"2024-04-23T14:34:21.025987+0000","last_undegraded":"2024-04-23T14:34:21.025987+0000","last_fullsized":"2024-04-23T14:34:21.025987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:16.223192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-23T14:34:21.027774+0000","last_change":"2024-04-23T14:34:21.026063+0000","last_active":"2024-04-23T14:34:21.027774+0000","last_peered":"2024-04-23T14:34:21.027774+0000","last_clean":"2024-04-23T14:34:21.027774+0000","last_became_active":"2024-04-23T14:34:21.025781+0000","last_became_peered":"2024-04-23T14:34:21.025781+0000","last_unstale":"2024-04-23T14:34:21.027774+0000","last_undegraded":"2024-04-23T14:34:21.027774+0000","last_fullsized":"2024-04-23T14:34:21.027774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:41:16.223192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830384,"statfs":{"total":107374182400,"available":107346313216,"internally_reserved":0,"allocated":200704,"data_stored":66933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830376,"statfs":{"total":107374182400,"available":107346305024,"internally_reserved":0,"allocated":200704,"data_stored":66933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":17,"seq":73014444035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830384,"statfs":{"total":107374182400,"available":107346313216,"internally_reserved":0,"allocated":200704,"data_stored":66933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":17,"seq":73014444036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27192,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830408,"statfs":{"total":107374182400,"available":107346337792,"internally_reserved":0,"allocated":176128,"data_stored":54653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,4,5,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830396,"statfs":{"total":107374182400,"available":107346325504,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476739,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":188416,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,5,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830452,"statfs":{"total":107374182400,"available":107346382848,"internally_reserved":0,"allocated":131072,"data_stored":29503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:34:26.567 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T14:34:26.567 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T14:34:26.568 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-23T14:34:26.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-23T14:34:26.876 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:34:26.876 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T14:34:26.894 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-23T14:34:26.894 INFO:teuthology.run_tasks:Running task workunit... 2024-04-23T14:34:26.908 INFO:tasks.workunit:Pulling workunits from ref 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T14:34:26.908 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-23T14:34:26.909 INFO:tasks.workunit:timeout=3h 2024-04-23T14:34:26.909 INFO:tasks.workunit:cleanup=True 2024-04-23T14:34:26.909 DEBUG:teuthology.orchestra.run.smithi084:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-23T14:34:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:34:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-23T14:34:26.914 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-23T14:34:26.960 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-23T14:34:26.960 DEBUG:teuthology.orchestra.run.smithi084:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-23T14:34:27.008 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5d349943c59c9485df060d6adb0594f3940ec0eb 2024-04-23T14:34:27.057 INFO:tasks.workunit.client.0.smithi084.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-23T14:35:17.112 INFO:tasks.workunit.client.0.smithi084.stderr:Note: switching to '5d349943c59c9485df060d6adb0594f3940ec0eb'. 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr: 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr:state without impacting any branches by switching back to a branch. 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr: 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr: 2024-04-23T14:35:17.113 INFO:tasks.workunit.client.0.smithi084.stderr: git switch -c 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr: 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr:Or undo this operation with: 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr: 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr: git switch - 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr: 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr: 2024-04-23T14:35:17.114 INFO:tasks.workunit.client.0.smithi084.stderr:HEAD is now at 5d349943c59 Merge branch 'wlk-dev' of https://github.com/wanglinke521/ceph into wip-yuri3-testing-2024-04-22-1115 2024-04-23T14:35:17.119 DEBUG:teuthology.orchestra.run.smithi084:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-23T14:35:17.165 INFO:tasks.workunit.client.0.smithi084.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-23T14:35:17.168 INFO:tasks.workunit.client.0.smithi084.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T14:35:17.168 INFO:tasks.workunit.client.0.smithi084.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-23T14:35:17.967 INFO:tasks.workunit.client.0.smithi084.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-23T14:35:18.036 INFO:tasks.workunit.client.0.smithi084.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-23T14:35:18.087 INFO:tasks.workunit.client.0.smithi084.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T14:35:18.089 INFO:tasks.workunit.client.0.smithi084.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T14:35:18.089 INFO:tasks.workunit.client.0.smithi084.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-23T14:35:18.132 INFO:tasks.workunit.client.0.smithi084.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T14:35:18.135 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-23T14:35:18.135 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-23T14:35:18.183 INFO:tasks.workunit:Running workunits matching rados/test_health_warnings.sh on client.0... 2024-04-23T14:35:18.185 INFO:tasks.workunit:Running workunit rados/test_health_warnings.sh... 2024-04-23T14:35:18.185 DEBUG:teuthology.orchestra.run.smithi084:workunit test rados/test_health_warnings.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5d349943c59c9485df060d6adb0594f3940ec0eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2024-04-23T14:35:18.240 INFO:tasks.workunit.client.0.smithi084.stderr:+ crushtool -o crushmap --build --num_osds 10 host straw 2 rack straw 2 row straw 2 root straw 0 2024-04-23T14:35:18.260 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd setcrushmap -i crushmap 2024-04-23T14:35:20.476 INFO:tasks.workunit.client.0.smithi084.stderr:7 2024-04-23T14:35:20.509 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd tree 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout:-11 10.00000 root root 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: -9 8.00000 row row0 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: -6 4.00000 rack rack0 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: -1 2.00000 host host0 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: 0 1.00000 osd.0 up 1.00000 1.00000 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: 1 1.00000 osd.1 up 1.00000 1.00000 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: -2 2.00000 host host1 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: 2 1.00000 osd.2 up 1.00000 1.00000 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: 3 1.00000 osd.3 up 1.00000 1.00000 2024-04-23T14:35:20.895 INFO:tasks.workunit.client.0.smithi084.stdout: -7 4.00000 rack rack1 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: -3 2.00000 host host2 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: 4 1.00000 osd.4 up 1.00000 1.00000 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: 5 1.00000 osd.5 up 1.00000 1.00000 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: -4 2.00000 host host3 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: 6 1.00000 osd.6 up 1.00000 1.00000 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: 7 1.00000 osd.7 up 1.00000 1.00000 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout:-10 2.00000 row row1 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: -8 2.00000 rack rack2 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: -5 2.00000 host host4 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: 8 1.00000 osd.8 up 1.00000 1.00000 2024-04-23T14:35:20.896 INFO:tasks.workunit.client.0.smithi084.stdout: 9 1.00000 osd.9 up 1.00000 1.00000 2024-04-23T14:35:20.911 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph tell 'osd.*' injectargs --osd_max_markdown_count 1024 --osd_max_markdown_period 1 2024-04-23T14:35:21.057 INFO:tasks.workunit.client.0.smithi084.stdout:osd.0: {} 2024-04-23T14:35:21.087 INFO:tasks.workunit.client.0.smithi084.stdout:osd.1: {} 2024-04-23T14:35:21.117 INFO:tasks.workunit.client.0.smithi084.stdout:osd.2: {} 2024-04-23T14:35:21.152 INFO:tasks.workunit.client.0.smithi084.stdout:osd.3: {} 2024-04-23T14:35:21.182 INFO:tasks.workunit.client.0.smithi084.stdout:osd.4: {} 2024-04-23T14:35:21.211 INFO:tasks.workunit.client.0.smithi084.stdout:osd.5: {} 2024-04-23T14:35:21.248 INFO:tasks.workunit.client.0.smithi084.stdout:osd.6: {} 2024-04-23T14:35:21.279 INFO:tasks.workunit.client.0.smithi084.stdout:osd.7: {} 2024-04-23T14:35:21.310 INFO:tasks.workunit.client.0.smithi084.stdout:osd.8: {} 2024-04-23T14:35:21.341 INFO:tasks.workunit.client.0.smithi084.stdout:osd.9: {} 2024-04-23T14:35:21.357 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd set noout 2024-04-23T14:35:23.484 INFO:tasks.workunit.client.0.smithi084.stderr:noout is set 2024-04-23T14:35:23.508 INFO:tasks.workunit.client.0.smithi084.stderr:+ test_mark_two_osds_same_host_down 2024-04-23T14:35:23.508 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd set noup 2024-04-23T14:35:25.490 INFO:tasks.workunit.client.0.smithi084.stderr:noup is set 2024-04-23T14:35:25.514 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd down osd.0 osd.1 2024-04-23T14:35:25.821 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:25.818+0000 7f47f54fd640 -1 mon.a@0(leader).osd e27 definitely_dead 0 2024-04-23T14:35:26.608 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:26.602+0000 7f47f54fd640 -1 mon.a@0(leader).osd e28 definitely_dead 0 2024-04-23T14:35:26.608 INFO:tasks.workunit.client.0.smithi084.stderr:osd.0 is already down. osd.1 is already down. 2024-04-23T14:35:26.626 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:27.107 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-04-23T14:35:27.107 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-04-23T14:35:27.107 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_DOWN: 2 osds down 2024-04-23T14:35:27.107 INFO:tasks.workunit.client.0.smithi084.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:27.107 INFO:tasks.workunit.client.0.smithi084.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:27.107 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-04-23T14:35:27.107 INFO:tasks.workunit.client.0.smithi084.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-04-23T14:35:27.123 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '1 host' 2024-04-23T14:35:27.123 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:27.579 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-04-23T14:35:27.579 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:27.579 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '2 osds' 2024-04-23T14:35:27.876 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:35:27.874+0000 7fa956cce640 -1 osd.1 29 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:28.051 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-04-23T14:35:28.052 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:28.052 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep osd.0 2024-04-23T14:35:28.515 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:35:28.510+0000 7fa9462cf640 -1 osd.1 30 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:28.554 INFO:tasks.workunit.client.0.smithi084.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:28.555 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:28.555 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep osd.1 2024-04-23T14:35:29.040 INFO:tasks.workunit.client.0.smithi084.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:29.040 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd unset noup 2024-04-23T14:35:29.527 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:35:29.522+0000 7fa94e2f0640 -1 osd.1 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:30.020 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:35:30.018+0000 7f517d0aa640 -1 osd.0 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:30.523 INFO:tasks.workunit.client.0.smithi084.stderr:noup is unset 2024-04-23T14:35:30.549 INFO:tasks.workunit.client.0.smithi084.stderr:+ wait_for_healthy 2024-04-23T14:35:30.549 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:30.550 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep down 2024-04-23T14:35:31.019 INFO:tasks.workunit.client.0.smithi084.stderr:+ test_mark_two_osds_same_rack_down 2024-04-23T14:35:31.019 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd set noup 2024-04-23T14:35:32.537 INFO:tasks.workunit.client.0.smithi084.stderr:noup is set 2024-04-23T14:35:32.562 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd down osd.8 osd.9 2024-04-23T14:35:32.849 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:32.846+0000 7f47f54fd640 -1 mon.a@0(leader).osd e34 definitely_dead 0 2024-04-23T14:35:33.640 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:33.638+0000 7f47f54fd640 -1 mon.a@0(leader).osd e35 definitely_dead 0 2024-04-23T14:35:33.641 INFO:tasks.workunit.client.0.smithi084.stderr:osd.8 is already down. osd.9 is already down. 2024-04-23T14:35:33.657 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:34.148 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-04-23T14:35:34.148 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-04-23T14:35:34.148 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_DOWN: 2 osds down 2024-04-23T14:35:34.148 INFO:tasks.workunit.client.0.smithi084.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-04-23T14:35:34.148 INFO:tasks.workunit.client.0.smithi084.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-04-23T14:35:34.149 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-04-23T14:35:34.149 INFO:tasks.workunit.client.0.smithi084.stdout: host host4 (root=root,row=row1,rack=rack2) (2 osds) is down 2024-04-23T14:35:34.149 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_RACK_DOWN: 1 rack (2 osds) down 2024-04-23T14:35:34.149 INFO:tasks.workunit.client.0.smithi084.stdout: rack rack2 (root=root,row=row1) (2 osds) is down 2024-04-23T14:35:34.149 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_ROW_DOWN: 1 row (2 osds) down 2024-04-23T14:35:34.149 INFO:tasks.workunit.client.0.smithi084.stdout: row row1 (root=root) (2 osds) is down 2024-04-23T14:35:34.164 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:34.164 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '1 host' 2024-04-23T14:35:34.606 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-04-23T14:35:34.606 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:34.607 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '1 rack' 2024-04-23T14:35:35.054 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-04-23T14:35:35.054 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:35.054 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '1 row' 2024-04-23T14:35:35.525 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-04-23T14:35:35.525 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:35.526 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '2 osds' 2024-04-23T14:35:35.989 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-04-23T14:35:35.990 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:35.990 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep osd.8 2024-04-23T14:35:36.427 INFO:tasks.ceph.osd.9.smithi084.stderr:2024-04-23T14:35:36.422+0000 7fa3d41f2640 -1 osd.9 37 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:36.476 INFO:tasks.workunit.client.0.smithi084.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-04-23T14:35:36.476 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:36.477 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep osd.9 2024-04-23T14:35:36.556 INFO:tasks.ceph.osd.9.smithi084.stderr:2024-04-23T14:35:36.554+0000 7fa3c77fb640 -1 osd.9 38 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:36.972 INFO:tasks.workunit.client.0.smithi084.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-04-23T14:35:36.972 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd unset noup 2024-04-23T14:35:37.126 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:35:37.122+0000 7f9589a87640 -1 osd.8 38 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:37.568 INFO:tasks.ceph.osd.9.smithi084.stderr:2024-04-23T14:35:37.562+0000 7fa3cf81c640 -1 osd.9 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:37.570 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:35:37.562+0000 7f95850b1640 -1 osd.8 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:38.562 INFO:tasks.workunit.client.0.smithi084.stderr:noup is unset 2024-04-23T14:35:38.587 INFO:tasks.workunit.client.0.smithi084.stderr:+ wait_for_healthy 2024-04-23T14:35:38.588 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:38.588 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep down 2024-04-23T14:35:39.056 INFO:tasks.workunit.client.0.smithi084.stderr:+ test_mark_all_but_last_osds_down 2024-04-23T14:35:39.056 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd set noup 2024-04-23T14:35:40.581 INFO:tasks.workunit.client.0.smithi084.stderr:noup is set 2024-04-23T14:35:40.606 INFO:tasks.workunit.client.0.smithi084.stderr:++ ceph osd ls 2024-04-23T14:35:40.606 INFO:tasks.workunit.client.0.smithi084.stderr:++ sed '$d' 2024-04-23T14:35:41.009 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd down 0 1 2 3 4 5 6 7 8 2024-04-23T14:35:41.320 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:41.318+0000 7f47f54fd640 -1 mon.a@0(leader).osd e42 definitely_dead 0 2024-04-23T14:35:41.700 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:41.694+0000 7f47f54fd640 -1 mon.a@0(leader).osd e43 definitely_dead 0 2024-04-23T14:35:41.700 INFO:tasks.workunit.client.0.smithi084.stderr:osd.0 is already down. osd.1 is already down. osd.2 is already down. osd.3 is already down. osd.4 is already down. osd.5 is already down. osd.6 is already down. osd.7 is already down. osd.8 is already down. 2024-04-23T14:35:41.717 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:42.151 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:35:42.146+0000 7f9589275640 -1 osd.8 43 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_DOWN: 9 osds down 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout: osd.2 (root=root,row=row0,rack=rack0,host=host1) is down 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout: osd.3 (root=root,row=row0,rack=rack0,host=host1) is down 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout: osd.4 (root=root,row=row0,rack=rack1,host=host2) is down 2024-04-23T14:35:42.221 INFO:tasks.workunit.client.0.smithi084.stdout: osd.5 (root=root,row=row0,rack=rack1,host=host2) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: osd.6 (root=root,row=row0,rack=rack1,host=host3) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: osd.7 (root=root,row=row0,rack=rack1,host=host3) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_HOST_DOWN: 4 hosts (8 osds) down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: host host1 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: host host2 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: host host3 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_RACK_DOWN: 2 racks (8 osds) down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: rack rack0 (root=root,row=row0) (4 osds) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: rack rack1 (root=root,row=row0) (4 osds) is down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_ROW_DOWN: 1 row (8 osds) down 2024-04-23T14:35:42.222 INFO:tasks.workunit.client.0.smithi084.stdout: row row0 (root=root) (8 osds) is down 2024-04-23T14:35:42.237 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:42.237 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '1 row' 2024-04-23T14:35:42.378 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:35:42.374+0000 7f3af7b3a640 -1 osd.6 43 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:42.459 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:35:42.454+0000 7fb560bc9640 -1 osd.5 43 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:42.601 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:35:42.598+0000 7f3ae794d640 -1 osd.6 44 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:42.601 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:35:42.598+0000 7fb5541d2640 -1 osd.5 44 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:42.601 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:35:42.598+0000 7f957d090640 -1 osd.8 44 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:42.689 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-04-23T14:35:42.689 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:42.689 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '2 racks' 2024-04-23T14:35:42.817 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-04-23T14:35:42.814+0000 7fa9fbf0c640 -1 osd.7 44 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:42.840 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:35:42.834+0000 7fa9564bc640 -1 osd.1 44 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.072 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:35:43.070+0000 7f518126e640 -1 osd.0 44 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.159 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-04-23T14:35:43.159 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:43.160 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '4 hosts' 2024-04-23T14:35:43.216 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-04-23T14:35:43.210+0000 7fab33f09640 -1 osd.3 44 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.602 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:35:43.598+0000 7f5175089640 -1 osd.0 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.602 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:35:43.598+0000 7f3ae794d640 -1 osd.6 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.602 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:35:43.598+0000 7fa9462cf640 -1 osd.1 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.603 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-04-23T14:35:43.598+0000 7fab2350a640 -1 osd.3 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.603 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:35:43.598+0000 7fb5541d2640 -1 osd.5 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.603 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-04-23T14:35:43.598+0000 7fa9eb50d640 -1 osd.7 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.603 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:35:43.598+0000 7f957d090640 -1 osd.8 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:43.613 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-04-23T14:35:43.614 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:43.614 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '9 osds' 2024-04-23T14:35:43.719 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-04-23T14:35:43.714+0000 7f6ea28f3640 -1 osd.4 45 osdmap NOUP flag is set, waiting for it to clear 2024-04-23T14:35:44.047 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-04-23T14:35:44.048 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd unset noup 2024-04-23T14:35:44.608 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-04-23T14:35:44.602+0000 7fab2b52b640 -1 osd.3 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:44.608 INFO:tasks.ceph.osd.7.smithi084.stderr:2024-04-23T14:35:44.602+0000 7fa9f352e640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:44.609 INFO:tasks.ceph.osd.6.smithi084.stderr:2024-04-23T14:35:44.606+0000 7f3aef96e640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:44.609 INFO:tasks.ceph.osd.5.smithi084.stderr:2024-04-23T14:35:44.606+0000 7fb55c1f3640 -1 osd.5 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:44.611 INFO:tasks.ceph.osd.8.smithi084.stderr:2024-04-23T14:35:44.606+0000 7f95850b1640 -1 osd.8 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:44.614 INFO:tasks.ceph.osd.4.smithi084.stderr:2024-04-23T14:35:44.610+0000 7f6e9df1d640 -1 osd.4 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:44.614 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:35:44.610+0000 7fa94e2f0640 -1 osd.1 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:44.614 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:35:44.610+0000 7f517d0aa640 -1 osd.0 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:45.045 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-04-23T14:35:45.042+0000 7f39ab160640 -1 osd.2 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:45.606 INFO:tasks.workunit.client.0.smithi084.stderr:noup is unset 2024-04-23T14:35:45.647 INFO:tasks.workunit.client.0.smithi084.stderr:+ wait_for_healthy 2024-04-23T14:35:45.648 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep down 2024-04-23T14:35:45.648 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:46.122 INFO:tasks.workunit.client.0.smithi084.stderr:+ test_mark_two_osds_same_host_down_with_classes 2024-04-23T14:35:46.122 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd set noup 2024-04-23T14:35:47.626 INFO:tasks.workunit.client.0.smithi084.stderr:noup is set 2024-04-23T14:35:47.650 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd crush set-device-class ssd osd.0 osd.2 osd.4 osd.6 osd.8 2024-04-23T14:35:49.633 INFO:tasks.workunit.client.0.smithi084.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. osd.2 already set to class ssd. set-device-class item id 2 name 'osd.2' device_class 'ssd': no change. osd.4 already set to class ssd. set-device-class item id 4 name 'osd.4' device_class 'ssd': no change. osd.6 already set to class ssd. set-device-class item id 6 name 'osd.6' device_class 'ssd': no change. osd.8 already set to class ssd. set-device-class item id 8 name 'osd.8' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-04-23T14:35:49.658 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd crush set-device-class hdd osd.1 osd.3 osd.5 osd.7 osd.9 2024-04-23T14:35:51.651 INFO:tasks.workunit.client.0.smithi084.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. osd.3 already set to class hdd. set-device-class item id 3 name 'osd.3' device_class 'hdd': no change. osd.5 already set to class hdd. set-device-class item id 5 name 'osd.5' device_class 'hdd': no change. osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. osd.9 already set to class hdd. set-device-class item id 9 name 'osd.9' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-04-23T14:35:51.672 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd down osd.0 osd.1 2024-04-23T14:35:51.967 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:51.961+0000 7f47f54fd640 -1 mon.a@0(leader).osd e53 definitely_dead 0 2024-04-23T14:35:52.771 INFO:tasks.ceph.mon.a.smithi084.stderr:2024-04-23T14:35:52.765+0000 7f47f54fd640 -1 mon.a@0(leader).osd e54 definitely_dead 0 2024-04-23T14:35:52.771 INFO:tasks.workunit.client.0.smithi084.stderr:osd.0 is already down. osd.1 is already down. 2024-04-23T14:35:52.789 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:53.287 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-04-23T14:35:53.287 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-04-23T14:35:53.287 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_DOWN: 2 osds down 2024-04-23T14:35:53.287 INFO:tasks.workunit.client.0.smithi084.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:53.287 INFO:tasks.workunit.client.0.smithi084.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:53.287 INFO:tasks.workunit.client.0.smithi084.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-04-23T14:35:53.287 INFO:tasks.workunit.client.0.smithi084.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-04-23T14:35:53.302 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:53.302 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '1 host' 2024-04-23T14:35:53.756 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-04-23T14:35:53.756 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:53.756 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep '2 osds' 2024-04-23T14:35:54.201 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-04-23T14:35:54.202 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep osd.0 2024-04-23T14:35:54.202 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:54.719 INFO:tasks.workunit.client.0.smithi084.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:54.719 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep osd.1 2024-04-23T14:35:54.719 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health detail 2024-04-23T14:35:55.211 INFO:tasks.workunit.client.0.smithi084.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-04-23T14:35:55.212 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph osd unset noup 2024-04-23T14:35:55.812 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-04-23T14:35:55.805+0000 7fa94e2f0640 -1 osd.1 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:56.685 INFO:tasks.workunit.client.0.smithi084.stderr:noup is unset 2024-04-23T14:35:56.711 INFO:tasks.workunit.client.0.smithi084.stderr:+ wait_for_healthy 2024-04-23T14:35:56.711 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:56.711 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep down 2024-04-23T14:35:57.193 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2024-04-23T14:35:57.193 INFO:tasks.workunit.client.0.smithi084.stderr:+ sleep 1 2024-04-23T14:35:57.982 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-04-23T14:35:57.977+0000 7f517d0aa640 -1 osd.0 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:58.195 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:58.196 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep down 2024-04-23T14:35:58.678 INFO:tasks.workunit.client.0.smithi084.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2024-04-23T14:35:58.679 INFO:tasks.workunit.client.0.smithi084.stderr:+ sleep 1 2024-04-23T14:35:59.681 INFO:tasks.workunit.client.0.smithi084.stderr:+ ceph health 2024-04-23T14:35:59.681 INFO:tasks.workunit.client.0.smithi084.stderr:+ grep down 2024-04-23T14:36:00.161 INFO:tasks.workunit.client.0.smithi084.stderr:+ exit 0 2024-04-23T14:36:00.161 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-23T14:36:00.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-23T14:36:00.218 INFO:tasks.workunit:Stopping ['rados/test_health_warnings.sh'] on client.0... 2024-04-23T14:36:00.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-23T14:36:00.568 DEBUG:teuthology.parallel:result is None 2024-04-23T14:36:00.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-23T14:36:00.580 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-23T14:36:00.581 DEBUG:teuthology.orchestra.run.smithi084:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-23T14:36:00.729 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-23T14:36:00.729 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-23T14:36:00.738 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T14:36:00.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:01.045 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:01.063 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-23T14:36:00.399437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276336,"kb_used_data":6100,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299664,"statfs":{"total":1073741824000,"available":1073458855936,"internally_reserved":0,"allocated":6246400,"data_stored":2956738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.693424+0000","last_change":"2024-04-23T14:35:46.624991+0000","last_active":"2024-04-23T14:35:57.693424+0000","last_peered":"2024-04-23T14:35:57.693424+0000","last_clean":"2024-04-23T14:35:57.693424+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:35:57.693424+0000","last_undegraded":"2024-04-23T14:35:57.693424+0000","last_fullsized":"2024-04-23T14:35:57.693424+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.821259+0000","last_change":"2024-04-23T14:35:46.627449+0000","last_active":"2024-04-23T14:35:57.821259+0000","last_peered":"2024-04-23T14:35:57.821259+0000","last_clean":"2024-04-23T14:35:57.821259+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:35:57.821259+0000","last_undegraded":"2024-04-23T14:35:57.821259+0000","last_fullsized":"2024-04-23T14:35:57.821259+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.474897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+undersized+wait","last_fresh":"2024-04-23T14:35:53.670136+0000","last_change":"2024-04-23T14:35:53.670016+0000","last_active":"2024-04-23T14:35:53.670136+0000","last_peered":"2024-04-23T14:35:53.670136+0000","last_clean":"2024-04-23T14:35:52.121923+0000","last_became_active":"2024-04-23T14:35:53.670016+0000","last_became_peered":"2024-04-23T14:35:53.670016+0000","last_unstale":"2024-04-23T14:35:53.670136+0000","last_undegraded":"2024-04-23T14:35:53.670136+0000","last_fullsized":"2024-04-23T14:35:53.669268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.824731+0000","last_change":"2024-04-23T14:35:57.824731+0000","last_active":"2024-04-23T14:35:57.824731+0000","last_peered":"2024-04-23T14:35:57.824731+0000","last_clean":"2024-04-23T14:35:57.824731+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:35:57.824731+0000","last_undegraded":"2024-04-23T14:35:57.824731+0000","last_fullsized":"2024-04-23T14:35:57.824731+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.697394+0000","last_change":"2024-04-23T14:35:57.697394+0000","last_active":"2024-04-23T14:35:57.697394+0000","last_peered":"2024-04-23T14:35:57.697394+0000","last_clean":"2024-04-23T14:35:57.697394+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:35:57.697394+0000","last_undegraded":"2024-04-23T14:35:57.697394+0000","last_fullsized":"2024-04-23T14:35:57.697394+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:23.151575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":75,"reported_epoch":56,"state":"peering","last_fresh":"2024-04-23T14:35:57.007389+0000","last_change":"2024-04-23T14:35:57.008593+0000","last_active":"2024-04-23T14:35:57.008593+0000","last_peered":"2024-04-23T14:35:57.007389+0000","last_clean":"2024-04-23T14:35:52.123118+0000","last_became_active":"2024-04-23T14:35:53.674955+0000","last_became_peered":"2024-04-23T14:35:53.674955+0000","last_unstale":"2024-04-23T14:35:57.007389+0000","last_undegraded":"2024-04-23T14:35:57.007389+0000","last_fullsized":"2024-04-23T14:35:53.672847+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.822516+0000","last_change":"2024-04-23T14:35:46.624638+0000","last_active":"2024-04-23T14:35:57.822516+0000","last_peered":"2024-04-23T14:35:57.822516+0000","last_clean":"2024-04-23T14:35:57.822516+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:35:57.822516+0000","last_undegraded":"2024-04-23T14:35:57.822516+0000","last_fullsized":"2024-04-23T14:35:57.822516+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:43.276743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-23T14:35:56.092998+0000","last_change":"2024-04-23T14:35:46.622348+0000","last_active":"2024-04-23T14:35:56.092998+0000","last_peered":"2024-04-23T14:35:56.092998+0000","last_clean":"2024-04-23T14:35:56.092998+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:35:56.092998+0000","last_undegraded":"2024-04-23T14:35:56.092998+0000","last_fullsized":"2024-04-23T14:35:56.092998+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:43:58.676638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462934,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":630784,"data_stored":299447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829996,"statfs":{"total":107374182400,"available":107345915904,"internally_reserved":0,"allocated":593920,"data_stored":276357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:01.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:01.359 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:01.377 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-23T14:36:00.399437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276336,"kb_used_data":6100,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299664,"statfs":{"total":1073741824000,"available":1073458855936,"internally_reserved":0,"allocated":6246400,"data_stored":2956738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.693424+0000","last_change":"2024-04-23T14:35:46.624991+0000","last_active":"2024-04-23T14:35:57.693424+0000","last_peered":"2024-04-23T14:35:57.693424+0000","last_clean":"2024-04-23T14:35:57.693424+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:35:57.693424+0000","last_undegraded":"2024-04-23T14:35:57.693424+0000","last_fullsized":"2024-04-23T14:35:57.693424+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.821259+0000","last_change":"2024-04-23T14:35:46.627449+0000","last_active":"2024-04-23T14:35:57.821259+0000","last_peered":"2024-04-23T14:35:57.821259+0000","last_clean":"2024-04-23T14:35:57.821259+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:35:57.821259+0000","last_undegraded":"2024-04-23T14:35:57.821259+0000","last_fullsized":"2024-04-23T14:35:57.821259+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.474897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+undersized+wait","last_fresh":"2024-04-23T14:35:53.670136+0000","last_change":"2024-04-23T14:35:53.670016+0000","last_active":"2024-04-23T14:35:53.670136+0000","last_peered":"2024-04-23T14:35:53.670136+0000","last_clean":"2024-04-23T14:35:52.121923+0000","last_became_active":"2024-04-23T14:35:53.670016+0000","last_became_peered":"2024-04-23T14:35:53.670016+0000","last_unstale":"2024-04-23T14:35:53.670136+0000","last_undegraded":"2024-04-23T14:35:53.670136+0000","last_fullsized":"2024-04-23T14:35:53.669268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.824731+0000","last_change":"2024-04-23T14:35:57.824731+0000","last_active":"2024-04-23T14:35:57.824731+0000","last_peered":"2024-04-23T14:35:57.824731+0000","last_clean":"2024-04-23T14:35:57.824731+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:35:57.824731+0000","last_undegraded":"2024-04-23T14:35:57.824731+0000","last_fullsized":"2024-04-23T14:35:57.824731+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.697394+0000","last_change":"2024-04-23T14:35:57.697394+0000","last_active":"2024-04-23T14:35:57.697394+0000","last_peered":"2024-04-23T14:35:57.697394+0000","last_clean":"2024-04-23T14:35:57.697394+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:35:57.697394+0000","last_undegraded":"2024-04-23T14:35:57.697394+0000","last_fullsized":"2024-04-23T14:35:57.697394+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:23.151575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":75,"reported_epoch":56,"state":"peering","last_fresh":"2024-04-23T14:35:57.007389+0000","last_change":"2024-04-23T14:35:57.008593+0000","last_active":"2024-04-23T14:35:57.008593+0000","last_peered":"2024-04-23T14:35:57.007389+0000","last_clean":"2024-04-23T14:35:52.123118+0000","last_became_active":"2024-04-23T14:35:53.674955+0000","last_became_peered":"2024-04-23T14:35:53.674955+0000","last_unstale":"2024-04-23T14:35:57.007389+0000","last_undegraded":"2024-04-23T14:35:57.007389+0000","last_fullsized":"2024-04-23T14:35:53.672847+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.822516+0000","last_change":"2024-04-23T14:35:46.624638+0000","last_active":"2024-04-23T14:35:57.822516+0000","last_peered":"2024-04-23T14:35:57.822516+0000","last_clean":"2024-04-23T14:35:57.822516+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:35:57.822516+0000","last_undegraded":"2024-04-23T14:35:57.822516+0000","last_fullsized":"2024-04-23T14:35:57.822516+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:43.276743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-23T14:35:56.092998+0000","last_change":"2024-04-23T14:35:46.622348+0000","last_active":"2024-04-23T14:35:56.092998+0000","last_peered":"2024-04-23T14:35:56.092998+0000","last_clean":"2024-04-23T14:35:56.092998+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:35:56.092998+0000","last_undegraded":"2024-04-23T14:35:56.092998+0000","last_fullsized":"2024-04-23T14:35:56.092998+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:43:58.676638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462934,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":630784,"data_stored":299447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829996,"statfs":{"total":107374182400,"available":107345915904,"internally_reserved":0,"allocated":593920,"data_stored":276357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:01.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:01.654 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:01.655 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:01.671 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-23T14:36:00.399437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276336,"kb_used_data":6100,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299664,"statfs":{"total":1073741824000,"available":1073458855936,"internally_reserved":0,"allocated":6246400,"data_stored":2956738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.693424+0000","last_change":"2024-04-23T14:35:46.624991+0000","last_active":"2024-04-23T14:35:57.693424+0000","last_peered":"2024-04-23T14:35:57.693424+0000","last_clean":"2024-04-23T14:35:57.693424+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:35:57.693424+0000","last_undegraded":"2024-04-23T14:35:57.693424+0000","last_fullsized":"2024-04-23T14:35:57.693424+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.821259+0000","last_change":"2024-04-23T14:35:46.627449+0000","last_active":"2024-04-23T14:35:57.821259+0000","last_peered":"2024-04-23T14:35:57.821259+0000","last_clean":"2024-04-23T14:35:57.821259+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:35:57.821259+0000","last_undegraded":"2024-04-23T14:35:57.821259+0000","last_fullsized":"2024-04-23T14:35:57.821259+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.474897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+undersized+wait","last_fresh":"2024-04-23T14:35:53.670136+0000","last_change":"2024-04-23T14:35:53.670016+0000","last_active":"2024-04-23T14:35:53.670136+0000","last_peered":"2024-04-23T14:35:53.670136+0000","last_clean":"2024-04-23T14:35:52.121923+0000","last_became_active":"2024-04-23T14:35:53.670016+0000","last_became_peered":"2024-04-23T14:35:53.670016+0000","last_unstale":"2024-04-23T14:35:53.670136+0000","last_undegraded":"2024-04-23T14:35:53.670136+0000","last_fullsized":"2024-04-23T14:35:53.669268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.824731+0000","last_change":"2024-04-23T14:35:57.824731+0000","last_active":"2024-04-23T14:35:57.824731+0000","last_peered":"2024-04-23T14:35:57.824731+0000","last_clean":"2024-04-23T14:35:57.824731+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:35:57.824731+0000","last_undegraded":"2024-04-23T14:35:57.824731+0000","last_fullsized":"2024-04-23T14:35:57.824731+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.697394+0000","last_change":"2024-04-23T14:35:57.697394+0000","last_active":"2024-04-23T14:35:57.697394+0000","last_peered":"2024-04-23T14:35:57.697394+0000","last_clean":"2024-04-23T14:35:57.697394+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:35:57.697394+0000","last_undegraded":"2024-04-23T14:35:57.697394+0000","last_fullsized":"2024-04-23T14:35:57.697394+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:23.151575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":75,"reported_epoch":56,"state":"peering","last_fresh":"2024-04-23T14:35:57.007389+0000","last_change":"2024-04-23T14:35:57.008593+0000","last_active":"2024-04-23T14:35:57.008593+0000","last_peered":"2024-04-23T14:35:57.007389+0000","last_clean":"2024-04-23T14:35:52.123118+0000","last_became_active":"2024-04-23T14:35:53.674955+0000","last_became_peered":"2024-04-23T14:35:53.674955+0000","last_unstale":"2024-04-23T14:35:57.007389+0000","last_undegraded":"2024-04-23T14:35:57.007389+0000","last_fullsized":"2024-04-23T14:35:53.672847+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.822516+0000","last_change":"2024-04-23T14:35:46.624638+0000","last_active":"2024-04-23T14:35:57.822516+0000","last_peered":"2024-04-23T14:35:57.822516+0000","last_clean":"2024-04-23T14:35:57.822516+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:35:57.822516+0000","last_undegraded":"2024-04-23T14:35:57.822516+0000","last_fullsized":"2024-04-23T14:35:57.822516+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:43.276743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-23T14:35:56.092998+0000","last_change":"2024-04-23T14:35:46.622348+0000","last_active":"2024-04-23T14:35:56.092998+0000","last_peered":"2024-04-23T14:35:56.092998+0000","last_clean":"2024-04-23T14:35:56.092998+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:35:56.092998+0000","last_undegraded":"2024-04-23T14:35:56.092998+0000","last_fullsized":"2024-04-23T14:35:56.092998+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:43:58.676638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462934,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":630784,"data_stored":299447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829996,"statfs":{"total":107374182400,"available":107345915904,"internally_reserved":0,"allocated":593920,"data_stored":276357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:01.672 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2024-04-23T14:36:01.672 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 84, 'reported_epoch': 55, 'state': 'active+undersized+wait', 'last_fresh': '2024-04-23T14:35:53.670136+0000', 'last_change': '2024-04-23T14:35:53.670016+0000', 'last_active': '2024-04-23T14:35:53.670136+0000', 'last_peered': '2024-04-23T14:35:53.670136+0000', 'last_clean': '2024-04-23T14:35:52.121923+0000', 'last_became_active': '2024-04-23T14:35:53.670016+0000', 'last_became_peered': '2024-04-23T14:35:53.670016+0000', 'last_unstale': '2024-04-23T14:35:53.670136+0000', 'last_undegraded': '2024-04-23T14:35:53.670136+0000', 'last_fullsized': '2024-04-23T14:35:53.669268+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T14:34:20.002943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T14:34:20.002943+0000', 'last_clean_scrub_stamp': '2024-04-23T14:34:20.002943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9], 'acting': [9], 'avail_no_missing': ['9'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-04-23T14:36:01.672 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2024-04-23T14:36:01.672 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "0'0", 'reported_seq': 75, 'reported_epoch': 56, 'state': 'peering', 'last_fresh': '2024-04-23T14:35:57.007389+0000', 'last_change': '2024-04-23T14:35:57.008593+0000', 'last_active': '2024-04-23T14:35:57.008593+0000', 'last_peered': '2024-04-23T14:35:57.007389+0000', 'last_clean': '2024-04-23T14:35:52.123118+0000', 'last_became_active': '2024-04-23T14:35:53.674955+0000', 'last_became_peered': '2024-04-23T14:35:53.674955+0000', 'last_unstale': '2024-04-23T14:35:57.007389+0000', 'last_undegraded': '2024-04-23T14:35:57.007389+0000', 'last_fullsized': '2024-04-23T14:35:53.672847+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-23T14:34:20.002943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-23T14:34:20.002943+0000', 'last_clean_scrub_stamp': '2024-04-23T14:34:20.002943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-23T14:36:01.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-23T14:36:01.998 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:01.999 INFO:teuthology.orchestra.run.smithi084.stdout:{"fsid":"75bd7860-6209-40b8-8227-0a2e97ca7f45","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":114,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":60,"num_osds":10,"num_up_osds":10,"osd_up_since":1713882958,"num_in_osds":10,"osd_in_since":1713882850,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+undersized+wait","count":1},{"state_name":"peering","count":1}],"num_pgs":8,"num_pools":1,"num_objects":0,"data_bytes":0,"bytes_used":282968064,"bytes_avail":1073458855936,"bytes_total":1073741824000,"inactive_pgs_ratio":0.125},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-23T14:35:20.389311+0000","services":{}},"progress_events":{}} 2024-04-23T14:36:02.016 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-23T14:36:02.016 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:02.304 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:02.304 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:02.323 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-23T14:36:00.399437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276336,"kb_used_data":6100,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299664,"statfs":{"total":1073741824000,"available":1073458855936,"internally_reserved":0,"allocated":6246400,"data_stored":2956738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.693424+0000","last_change":"2024-04-23T14:35:46.624991+0000","last_active":"2024-04-23T14:35:57.693424+0000","last_peered":"2024-04-23T14:35:57.693424+0000","last_clean":"2024-04-23T14:35:57.693424+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:35:57.693424+0000","last_undegraded":"2024-04-23T14:35:57.693424+0000","last_fullsized":"2024-04-23T14:35:57.693424+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.821259+0000","last_change":"2024-04-23T14:35:46.627449+0000","last_active":"2024-04-23T14:35:57.821259+0000","last_peered":"2024-04-23T14:35:57.821259+0000","last_clean":"2024-04-23T14:35:57.821259+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:35:57.821259+0000","last_undegraded":"2024-04-23T14:35:57.821259+0000","last_fullsized":"2024-04-23T14:35:57.821259+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.474897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":84,"reported_epoch":55,"state":"active+undersized+wait","last_fresh":"2024-04-23T14:35:53.670136+0000","last_change":"2024-04-23T14:35:53.670016+0000","last_active":"2024-04-23T14:35:53.670136+0000","last_peered":"2024-04-23T14:35:53.670136+0000","last_clean":"2024-04-23T14:35:52.121923+0000","last_became_active":"2024-04-23T14:35:53.670016+0000","last_became_peered":"2024-04-23T14:35:53.670016+0000","last_unstale":"2024-04-23T14:35:53.670136+0000","last_undegraded":"2024-04-23T14:35:53.670136+0000","last_fullsized":"2024-04-23T14:35:53.669268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.824731+0000","last_change":"2024-04-23T14:35:57.824731+0000","last_active":"2024-04-23T14:35:57.824731+0000","last_peered":"2024-04-23T14:35:57.824731+0000","last_clean":"2024-04-23T14:35:57.824731+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:35:57.824731+0000","last_undegraded":"2024-04-23T14:35:57.824731+0000","last_fullsized":"2024-04-23T14:35:57.824731+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.697394+0000","last_change":"2024-04-23T14:35:57.697394+0000","last_active":"2024-04-23T14:35:57.697394+0000","last_peered":"2024-04-23T14:35:57.697394+0000","last_clean":"2024-04-23T14:35:57.697394+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:35:57.697394+0000","last_undegraded":"2024-04-23T14:35:57.697394+0000","last_fullsized":"2024-04-23T14:35:57.697394+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:23.151575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":75,"reported_epoch":56,"state":"peering","last_fresh":"2024-04-23T14:35:57.007389+0000","last_change":"2024-04-23T14:35:57.008593+0000","last_active":"2024-04-23T14:35:57.008593+0000","last_peered":"2024-04-23T14:35:57.007389+0000","last_clean":"2024-04-23T14:35:52.123118+0000","last_became_active":"2024-04-23T14:35:53.674955+0000","last_became_peered":"2024-04-23T14:35:53.674955+0000","last_unstale":"2024-04-23T14:35:57.007389+0000","last_undegraded":"2024-04-23T14:35:57.007389+0000","last_fullsized":"2024-04-23T14:35:53.672847+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-23T14:35:57.822516+0000","last_change":"2024-04-23T14:35:46.624638+0000","last_active":"2024-04-23T14:35:57.822516+0000","last_peered":"2024-04-23T14:35:57.822516+0000","last_clean":"2024-04-23T14:35:57.822516+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:35:57.822516+0000","last_undegraded":"2024-04-23T14:35:57.822516+0000","last_fullsized":"2024-04-23T14:35:57.822516+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:43.276743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-23T14:35:56.092998+0000","last_change":"2024-04-23T14:35:46.622348+0000","last_active":"2024-04-23T14:35:56.092998+0000","last_peered":"2024-04-23T14:35:56.092998+0000","last_clean":"2024-04-23T14:35:56.092998+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:35:56.092998+0000","last_undegraded":"2024-04-23T14:35:56.092998+0000","last_fullsized":"2024-04-23T14:35:56.092998+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:43:58.676638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829972,"statfs":{"total":107374182400,"available":107345891328,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462934,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135895,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":655360,"data_stored":315184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829960,"statfs":{"total":107374182400,"available":107345879040,"internally_reserved":0,"allocated":630784,"data_stored":299447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829976,"statfs":{"total":107374182400,"available":107345895424,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27604,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829996,"statfs":{"total":107374182400,"available":107345915904,"internally_reserved":0,"allocated":593920,"data_stored":276357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829968,"statfs":{"total":107374182400,"available":107345887232,"internally_reserved":0,"allocated":618496,"data_stored":291761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:05.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:05.626 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:05.627 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:05.645 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-23T14:36:04.400607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276868,"kb_used_data":6640,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299132,"statfs":{"total":1073741824000,"available":1073458311168,"internally_reserved":0,"allocated":6799360,"data_stored":3308630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810949+0000","last_change":"2024-04-23T14:35:46.624991+0000","last_active":"2024-04-23T14:36:00.810949+0000","last_peered":"2024-04-23T14:36:00.810949+0000","last_clean":"2024-04-23T14:36:00.810949+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:36:00.810949+0000","last_undegraded":"2024-04-23T14:36:00.810949+0000","last_fullsized":"2024-04-23T14:36:00.810949+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.811875+0000","last_change":"2024-04-23T14:35:46.627449+0000","last_active":"2024-04-23T14:36:00.811875+0000","last_peered":"2024-04-23T14:36:00.811875+0000","last_clean":"2024-04-23T14:36:00.811875+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:36:00.811875+0000","last_undegraded":"2024-04-23T14:36:00.811875+0000","last_fullsized":"2024-04-23T14:36:00.811875+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.474897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":99,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.106596+0000","last_change":"2024-04-23T14:36:00.106596+0000","last_active":"2024-04-23T14:36:00.106596+0000","last_peered":"2024-04-23T14:36:00.106596+0000","last_clean":"2024-04-23T14:36:00.106596+0000","last_became_active":"2024-04-23T14:36:00.105906+0000","last_became_peered":"2024-04-23T14:36:00.105906+0000","last_unstale":"2024-04-23T14:36:00.106596+0000","last_undegraded":"2024-04-23T14:36:00.106596+0000","last_fullsized":"2024-04-23T14:36:00.106596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:06:56.146752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:35:59.735476+0000","last_change":"2024-04-23T14:35:57.824731+0000","last_active":"2024-04-23T14:35:59.735476+0000","last_peered":"2024-04-23T14:35:59.735476+0000","last_clean":"2024-04-23T14:35:59.735476+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:35:59.735476+0000","last_undegraded":"2024-04-23T14:35:59.735476+0000","last_fullsized":"2024-04-23T14:35:59.735476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":90,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810914+0000","last_change":"2024-04-23T14:35:57.697394+0000","last_active":"2024-04-23T14:36:00.810914+0000","last_peered":"2024-04-23T14:36:00.810914+0000","last_clean":"2024-04-23T14:36:00.810914+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:36:00.810914+0000","last_undegraded":"2024-04-23T14:36:00.810914+0000","last_fullsized":"2024-04-23T14:36:00.810914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:23.151575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":87,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.812004+0000","last_change":"2024-04-23T14:35:57.698126+0000","last_active":"2024-04-23T14:36:00.812004+0000","last_peered":"2024-04-23T14:36:00.812004+0000","last_clean":"2024-04-23T14:36:00.812004+0000","last_became_active":"2024-04-23T14:35:57.697166+0000","last_became_peered":"2024-04-23T14:35:57.697166+0000","last_unstale":"2024-04-23T14:36:00.812004+0000","last_undegraded":"2024-04-23T14:36:00.812004+0000","last_fullsized":"2024-04-23T14:36:00.812004+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:27:33.229466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:35:59.703561+0000","last_change":"2024-04-23T14:35:46.624638+0000","last_active":"2024-04-23T14:35:59.703561+0000","last_peered":"2024-04-23T14:35:59.703561+0000","last_clean":"2024-04-23T14:35:59.703561+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:35:59.703561+0000","last_undegraded":"2024-04-23T14:35:59.703561+0000","last_fullsized":"2024-04-23T14:35:59.703561+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:43.276743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":83,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810569+0000","last_change":"2024-04-23T14:35:46.622348+0000","last_active":"2024-04-23T14:36:00.810569+0000","last_peered":"2024-04-23T14:36:00.810569+0000","last_clean":"2024-04-23T14:36:00.810569+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:36:00.810569+0000","last_undegraded":"2024-04-23T14:36:00.810569+0000","last_fullsized":"2024-04-23T14:36:00.810569+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:43:58.676638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135896,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462936,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070488,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:05.646 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T14:36:05.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:05.934 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:05.934 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:05.951 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-23T14:36:04.400607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276868,"kb_used_data":6640,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299132,"statfs":{"total":1073741824000,"available":1073458311168,"internally_reserved":0,"allocated":6799360,"data_stored":3308630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810949+0000","last_change":"2024-04-23T14:35:46.624991+0000","last_active":"2024-04-23T14:36:00.810949+0000","last_peered":"2024-04-23T14:36:00.810949+0000","last_clean":"2024-04-23T14:36:00.810949+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:36:00.810949+0000","last_undegraded":"2024-04-23T14:36:00.810949+0000","last_fullsized":"2024-04-23T14:36:00.810949+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.811875+0000","last_change":"2024-04-23T14:35:46.627449+0000","last_active":"2024-04-23T14:36:00.811875+0000","last_peered":"2024-04-23T14:36:00.811875+0000","last_clean":"2024-04-23T14:36:00.811875+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:36:00.811875+0000","last_undegraded":"2024-04-23T14:36:00.811875+0000","last_fullsized":"2024-04-23T14:36:00.811875+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.474897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":99,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.106596+0000","last_change":"2024-04-23T14:36:00.106596+0000","last_active":"2024-04-23T14:36:00.106596+0000","last_peered":"2024-04-23T14:36:00.106596+0000","last_clean":"2024-04-23T14:36:00.106596+0000","last_became_active":"2024-04-23T14:36:00.105906+0000","last_became_peered":"2024-04-23T14:36:00.105906+0000","last_unstale":"2024-04-23T14:36:00.106596+0000","last_undegraded":"2024-04-23T14:36:00.106596+0000","last_fullsized":"2024-04-23T14:36:00.106596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:06:56.146752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:35:59.735476+0000","last_change":"2024-04-23T14:35:57.824731+0000","last_active":"2024-04-23T14:35:59.735476+0000","last_peered":"2024-04-23T14:35:59.735476+0000","last_clean":"2024-04-23T14:35:59.735476+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:35:59.735476+0000","last_undegraded":"2024-04-23T14:35:59.735476+0000","last_fullsized":"2024-04-23T14:35:59.735476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":90,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810914+0000","last_change":"2024-04-23T14:35:57.697394+0000","last_active":"2024-04-23T14:36:00.810914+0000","last_peered":"2024-04-23T14:36:00.810914+0000","last_clean":"2024-04-23T14:36:00.810914+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:36:00.810914+0000","last_undegraded":"2024-04-23T14:36:00.810914+0000","last_fullsized":"2024-04-23T14:36:00.810914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:23.151575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":87,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.812004+0000","last_change":"2024-04-23T14:35:57.698126+0000","last_active":"2024-04-23T14:36:00.812004+0000","last_peered":"2024-04-23T14:36:00.812004+0000","last_clean":"2024-04-23T14:36:00.812004+0000","last_became_active":"2024-04-23T14:35:57.697166+0000","last_became_peered":"2024-04-23T14:35:57.697166+0000","last_unstale":"2024-04-23T14:36:00.812004+0000","last_undegraded":"2024-04-23T14:36:00.812004+0000","last_fullsized":"2024-04-23T14:36:00.812004+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:27:33.229466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:35:59.703561+0000","last_change":"2024-04-23T14:35:46.624638+0000","last_active":"2024-04-23T14:35:59.703561+0000","last_peered":"2024-04-23T14:35:59.703561+0000","last_clean":"2024-04-23T14:35:59.703561+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:35:59.703561+0000","last_undegraded":"2024-04-23T14:35:59.703561+0000","last_fullsized":"2024-04-23T14:35:59.703561+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:43.276743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":83,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810569+0000","last_change":"2024-04-23T14:35:46.622348+0000","last_active":"2024-04-23T14:36:00.810569+0000","last_peered":"2024-04-23T14:36:00.810569+0000","last_clean":"2024-04-23T14:36:00.810569+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:36:00.810569+0000","last_undegraded":"2024-04-23T14:36:00.810569+0000","last_fullsized":"2024-04-23T14:36:00.810569+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:43:58.676638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135896,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462936,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070488,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:05.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:36:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:06.250 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":60,"fsid":"75bd7860-6209-40b8-8227-0a2e97ca7f45","created":"2024-04-23T14:34:07.470391+0000","modified":"2024-04-23T14:35:59.695614+0000","last_up_change":"2024-04-23T14:35:58.692297+0000","last_in_change":"2024-04-23T14:34:10.475939+0000","flags":"noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800064,"flags_set":["noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T14:34:19.601403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.80000001192092896,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51365f97-9e49-4fe3-88a1-882cb68532fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":58,"up_from":59,"up_thru":35,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6801","nonce":1294248214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":1297248214},{"type":"v1","addr":"172.21.15.84:6838","nonce":1297248214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6807","nonce":1294248214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":1294248214},{"type":"v1","addr":"172.21.15.84:6805","nonce":1294248214}]},"public_addr":"172.21.15.84:6801/1294248214","cluster_addr":"172.21.15.84:6838/1297248214","heartbeat_back_addr":"172.21.15.84:6807/1294248214","heartbeat_front_addr":"172.21.15.84:6805/1294248214","state":["exists","up"]},{"osd":1,"uuid":"b37a2dd3-7326-40ba-b467-aa87c65d003a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":56,"up_from":57,"up_thru":57,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6834","nonce":4126848561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":4129848561},{"type":"v1","addr":"172.21.15.84:6822","nonce":4129848561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6846","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6851","nonce":4126848561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":4126848561},{"type":"v1","addr":"172.21.15.84:6842","nonce":4126848561}]},"public_addr":"172.21.15.84:6834/4126848561","cluster_addr":"172.21.15.84:6822/4129848561","heartbeat_back_addr":"172.21.15.84:6851/4126848561","heartbeat_front_addr":"172.21.15.84:6842/4126848561","state":["exists","up"]},{"osd":2,"uuid":"28625c48-3473-4b1b-84c4-3958ce03b774","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":0,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6818","nonce":187511398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6852","nonce":188511398},{"type":"v1","addr":"172.21.15.84:6854","nonce":188511398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6830","nonce":187511398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":187511398},{"type":"v1","addr":"172.21.15.84:6826","nonce":187511398}]},"public_addr":"172.21.15.84:6818/187511398","cluster_addr":"172.21.15.84:6854/188511398","heartbeat_back_addr":"172.21.15.84:6830/187511398","heartbeat_front_addr":"172.21.15.84:6826/187511398","state":["exists","up"]},{"osd":3,"uuid":"8cb99e87-e23c-4955-ac67-d51837a04a22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":46,"up_from":47,"up_thru":57,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6817","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6819","nonce":2048464647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6837","nonce":2049464647},{"type":"v1","addr":"172.21.15.84:6839","nonce":2049464647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6829","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6831","nonce":2048464647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6825","nonce":2048464647},{"type":"v1","addr":"172.21.15.84:6827","nonce":2048464647}]},"public_addr":"172.21.15.84:6819/2048464647","cluster_addr":"172.21.15.84:6839/2049464647","heartbeat_back_addr":"172.21.15.84:6831/2048464647","heartbeat_front_addr":"172.21.15.84:6827/2048464647","state":["exists","up"]},{"osd":4,"uuid":"7caefffd-ba0b-4d2a-b5f9-a1c17b6d06f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":57,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6843","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6848","nonce":575087180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6882","nonce":576087180},{"type":"v1","addr":"172.21.15.84:6883","nonce":576087180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6858","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6860","nonce":575087180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6855","nonce":575087180},{"type":"v1","addr":"172.21.15.84:6857","nonce":575087180}]},"public_addr":"172.21.15.84:6848/575087180","cluster_addr":"172.21.15.84:6883/576087180","heartbeat_back_addr":"172.21.15.84:6860/575087180","heartbeat_front_addr":"172.21.15.84:6857/575087180","state":["exists","up"]},{"osd":5,"uuid":"20da4f0c-bdb8-40cb-9a04-823ae3be38eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":47,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6809","nonce":1453654031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6853","nonce":1454654031},{"type":"v1","addr":"172.21.15.84:6856","nonce":1454654031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6815","nonce":1453654031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":1453654031},{"type":"v1","addr":"172.21.15.84:6813","nonce":1453654031}]},"public_addr":"172.21.15.84:6809/1453654031","cluster_addr":"172.21.15.84:6856/1454654031","heartbeat_back_addr":"172.21.15.84:6815/1453654031","heartbeat_front_addr":"172.21.15.84:6813/1453654031","state":["exists","up"]},{"osd":6,"uuid":"0ca25a0e-435b-4dd9-b69d-d803cff62eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":46,"up_from":47,"up_thru":54,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6844","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6849","nonce":832042711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6865","nonce":833042711},{"type":"v1","addr":"172.21.15.84:6810","nonce":833042711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6864","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6866","nonce":832042711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6859","nonce":832042711},{"type":"v1","addr":"172.21.15.84:6862","nonce":832042711}]},"public_addr":"172.21.15.84:6849/832042711","cluster_addr":"172.21.15.84:6810/833042711","heartbeat_back_addr":"172.21.15.84:6866/832042711","heartbeat_front_addr":"172.21.15.84:6862/832042711","state":["exists","up"]},{"osd":7,"uuid":"63c997f5-c82a-404e-8a1c-35234156c1c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":46,"up_from":47,"up_thru":47,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6833","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6835","nonce":3079998612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3080998612},{"type":"v1","addr":"172.21.15.84:6867","nonce":3080998612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6847","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6850","nonce":3079998612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6841","nonce":3079998612},{"type":"v1","addr":"172.21.15.84:6845","nonce":3079998612}]},"public_addr":"172.21.15.84:6835/3079998612","cluster_addr":"172.21.15.84:6867/3080998612","heartbeat_back_addr":"172.21.15.84:6850/3079998612","heartbeat_front_addr":"172.21.15.84:6845/3079998612","state":["exists","up"]},{"osd":8,"uuid":"1a6b6c1c-7f60-4ede-9473-f401bac5c289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":19,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6872","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6873","nonce":1867330816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6874","nonce":1869330816},{"type":"v1","addr":"172.21.15.84:6875","nonce":1869330816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6878","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6879","nonce":1867330816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6876","nonce":1867330816},{"type":"v1","addr":"172.21.15.84:6877","nonce":1867330816}]},"public_addr":"172.21.15.84:6873/1867330816","cluster_addr":"172.21.15.84:6875/1869330816","heartbeat_back_addr":"172.21.15.84:6879/1867330816","heartbeat_front_addr":"172.21.15.84:6877/1867330816","state":["exists","up"]},{"osd":9,"uuid":"c3de37db-80eb-42f7-ad9e-e539d93ec164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":39,"up_from":40,"up_thru":59,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6861","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6863","nonce":3429805588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":3430805588},{"type":"v1","addr":"172.21.15.84:6803","nonce":3430805588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6870","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6871","nonce":3429805588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6868","nonce":3429805588},{"type":"v1","addr":"172.21.15.84:6869","nonce":3429805588}]},"public_addr":"172.21.15.84:6863/3429805588","cluster_addr":"172.21.15.84:6803/3430805588","heartbeat_back_addr":"172.21.15.84:6871/3429805588","heartbeat_front_addr":"172.21.15.84:6869/3429805588","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-04-23T14:35:52.656117+0000","laggy_probability":0.65700000524520874,"laggy_interval":2,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.009829+0000","dead_epoch":58},{"osd":1,"down_stamp":"2024-04-23T14:35:52.656117+0000","laggy_probability":0.65700000524520874,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.325729+0000","dead_epoch":55},{"osd":2,"down_stamp":"2024-04-23T14:35:41.581676+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:14.359911+0000","dead_epoch":46},{"osd":3,"down_stamp":"2024-04-23T14:35:41.581676+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.126904+0000","dead_epoch":43},{"osd":4,"down_stamp":"2024-04-23T14:35:41.581676+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.150634+0000","dead_epoch":44},{"osd":5,"down_stamp":"2024-04-23T14:35:41.581676+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.070198+0000","dead_epoch":43},{"osd":6,"down_stamp":"2024-04-23T14:35:41.581676+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.375482+0000","dead_epoch":43},{"osd":7,"down_stamp":"2024-04-23T14:35:41.581676+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.280610+0000","dead_epoch":43},{"osd":8,"down_stamp":"2024-04-23T14:35:41.581676+0000","laggy_probability":0.50999999046325684,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.331276+0000","dead_epoch":43},{"osd":9,"down_stamp":"2024-04-23T14:35:33.537239+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:34:15.372023+0000","dead_epoch":37}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:36:07.269 INFO:tasks.ceph:Scrubbing osd.0 2024-04-23T14:36:07.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:07.429 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:07.429 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:07.429 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:07.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-23T14:36:07.736 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 0 to deep-scrub 2024-04-23T14:36:07.754 INFO:tasks.ceph:Scrubbing osd.1 2024-04-23T14:36:07.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:07.901 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:07.902 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:07.902 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:07.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-23T14:36:08.194 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 1 to deep-scrub 2024-04-23T14:36:08.213 INFO:tasks.ceph:Scrubbing osd.2 2024-04-23T14:36:08.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:08.354 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:08.354 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:08.355 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:08.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-23T14:36:08.638 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 2 to deep-scrub 2024-04-23T14:36:08.656 INFO:tasks.ceph:Scrubbing osd.3 2024-04-23T14:36:08.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:08.803 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:08.803 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:08.803 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:08.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-23T14:36:09.107 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 3 to deep-scrub 2024-04-23T14:36:09.126 INFO:tasks.ceph:Scrubbing osd.4 2024-04-23T14:36:09.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:09.262 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:09.262 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:09.262 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:09.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-23T14:36:09.554 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 4 to deep-scrub 2024-04-23T14:36:09.572 INFO:tasks.ceph:Scrubbing osd.5 2024-04-23T14:36:09.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:09.706 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:09.706 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:09.707 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:09.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-23T14:36:09.990 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 5 to deep-scrub 2024-04-23T14:36:10.008 INFO:tasks.ceph:Scrubbing osd.6 2024-04-23T14:36:10.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:10.140 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:10.141 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:10.141 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:10.155 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-23T14:36:10.428 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 6 to deep-scrub 2024-04-23T14:36:10.445 INFO:tasks.ceph:Scrubbing osd.7 2024-04-23T14:36:10.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:10.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-23T14:36:10.890 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 7 to deep-scrub 2024-04-23T14:36:10.909 INFO:tasks.ceph:Scrubbing osd.8 2024-04-23T14:36:10.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:11.051 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:11.052 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:11.052 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:11.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-04-23T14:36:11.361 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 8 to deep-scrub 2024-04-23T14:36:11.379 INFO:tasks.ceph:Scrubbing osd.9 2024-04-23T14:36:11.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-04-23T14:36:11.522 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-23T14:36:11.522 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T14:36:11.522 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-23T14:36:11.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-04-23T14:36:11.813 INFO:teuthology.orchestra.run.smithi084.stderr:instructed osd(s) 9 to deep-scrub 2024-04-23T14:36:11.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:12.123 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:12.141 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-04-23T14:36:10.402048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276868,"kb_used_data":6640,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299132,"statfs":{"total":1073741824000,"available":1073458311168,"internally_reserved":0,"allocated":6799360,"data_stored":3308630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810949+0000","last_change":"2024-04-23T14:35:46.624991+0000","last_active":"2024-04-23T14:36:00.810949+0000","last_peered":"2024-04-23T14:36:00.810949+0000","last_clean":"2024-04-23T14:36:00.810949+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:36:00.810949+0000","last_undegraded":"2024-04-23T14:36:00.810949+0000","last_fullsized":"2024-04-23T14:36:00.810949+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:42:59.429432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.811875+0000","last_change":"2024-04-23T14:35:46.627449+0000","last_active":"2024-04-23T14:36:00.811875+0000","last_peered":"2024-04-23T14:36:00.811875+0000","last_clean":"2024-04-23T14:36:00.811875+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:36:00.811875+0000","last_undegraded":"2024-04-23T14:36:00.811875+0000","last_fullsized":"2024-04-23T14:36:00.811875+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:13:23.474897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":99,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.106596+0000","last_change":"2024-04-23T14:36:00.106596+0000","last_active":"2024-04-23T14:36:00.106596+0000","last_peered":"2024-04-23T14:36:00.106596+0000","last_clean":"2024-04-23T14:36:00.106596+0000","last_became_active":"2024-04-23T14:36:00.105906+0000","last_became_peered":"2024-04-23T14:36:00.105906+0000","last_unstale":"2024-04-23T14:36:00.106596+0000","last_undegraded":"2024-04-23T14:36:00.106596+0000","last_fullsized":"2024-04-23T14:36:00.106596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:06:56.146752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:35:59.735580+0000","last_change":"2024-04-23T14:35:57.824731+0000","last_active":"2024-04-23T14:35:59.735580+0000","last_peered":"2024-04-23T14:35:59.735580+0000","last_clean":"2024-04-23T14:35:59.735580+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:35:59.735580+0000","last_undegraded":"2024-04-23T14:35:59.735580+0000","last_fullsized":"2024-04-23T14:35:59.735580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":90,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810914+0000","last_change":"2024-04-23T14:35:57.697394+0000","last_active":"2024-04-23T14:36:00.810914+0000","last_peered":"2024-04-23T14:36:00.810914+0000","last_clean":"2024-04-23T14:36:00.810914+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:36:00.810914+0000","last_undegraded":"2024-04-23T14:36:00.810914+0000","last_fullsized":"2024-04-23T14:36:00.810914+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:46:23.151575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":87,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.812004+0000","last_change":"2024-04-23T14:35:57.698126+0000","last_active":"2024-04-23T14:36:00.812004+0000","last_peered":"2024-04-23T14:36:00.812004+0000","last_clean":"2024-04-23T14:36:00.812004+0000","last_became_active":"2024-04-23T14:35:57.697166+0000","last_became_peered":"2024-04-23T14:35:57.697166+0000","last_unstale":"2024-04-23T14:36:00.812004+0000","last_undegraded":"2024-04-23T14:36:00.812004+0000","last_fullsized":"2024-04-23T14:36:00.812004+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:27:33.229466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:35:59.703561+0000","last_change":"2024-04-23T14:35:46.624638+0000","last_active":"2024-04-23T14:35:59.703561+0000","last_peered":"2024-04-23T14:35:59.703561+0000","last_clean":"2024-04-23T14:35:59.703561+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:35:59.703561+0000","last_undegraded":"2024-04-23T14:35:59.703561+0000","last_fullsized":"2024-04-23T14:35:59.703561+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:43.276743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":83,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:00.810569+0000","last_change":"2024-04-23T14:35:46.622348+0000","last_active":"2024-04-23T14:36:00.810569+0000","last_peered":"2024-04-23T14:36:00.810569+0000","last_clean":"2024-04-23T14:36:00.810569+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:36:00.810569+0000","last_undegraded":"2024-04-23T14:36:00.810569+0000","last_fullsized":"2024-04-23T14:36:00.810569+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:34:20.002943+0000","last_clean_scrub_stamp":"2024-04-23T14:34:20.002943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:43:58.676638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462936,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462937,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070489,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-23T14:34:20.002943+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=34, tm_sec=20, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=14, tm_min=36, tm_sec=6, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T14:36:12.145 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-23T14:36:32.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:36:32.455 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-23T14:36:32.456 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-23T14:36:32.474 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-04-23T14:36:32.408602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":276824,"kb_used_data":6640,"kb_used_omap":15,"kb_used_meta":270064,"kb_avail":1048299176,"statfs":{"total":1073741824000,"available":1073458356224,"internally_reserved":0,"allocated":6799360,"data_stored":3308630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":276545973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:10.272048+0000","last_change":"2024-04-23T14:36:10.272082+0000","last_active":"2024-04-23T14:36:10.272048+0000","last_peered":"2024-04-23T14:36:10.272048+0000","last_clean":"2024-04-23T14:36:10.272048+0000","last_became_active":"2024-04-23T14:35:46.624256+0000","last_became_peered":"2024-04-23T14:35:46.624256+0000","last_unstale":"2024-04-23T14:36:10.272048+0000","last_undegraded":"2024-04-23T14:36:10.272048+0000","last_fullsized":"2024-04-23T14:36:10.272048+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:10.271984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:10.271984+0000","last_clean_scrub_stamp":"2024-04-23T14:36:10.271984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:22.595577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":96,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:10.591159+0000","last_change":"2024-04-23T14:36:10.591159+0000","last_active":"2024-04-23T14:36:10.591159+0000","last_peered":"2024-04-23T14:36:10.591159+0000","last_clean":"2024-04-23T14:36:10.591159+0000","last_became_active":"2024-04-23T14:35:46.626934+0000","last_became_peered":"2024-04-23T14:35:46.626934+0000","last_unstale":"2024-04-23T14:36:10.591159+0000","last_undegraded":"2024-04-23T14:36:10.591159+0000","last_fullsized":"2024-04-23T14:36:10.591159+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:10.591068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:10.591068+0000","last_clean_scrub_stamp":"2024-04-23T14:36:10.591068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:12:52.259595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:11.951313+0000","last_change":"2024-04-23T14:36:11.951336+0000","last_active":"2024-04-23T14:36:11.951313+0000","last_peered":"2024-04-23T14:36:11.951313+0000","last_clean":"2024-04-23T14:36:11.951313+0000","last_became_active":"2024-04-23T14:36:00.105906+0000","last_became_peered":"2024-04-23T14:36:00.105906+0000","last_unstale":"2024-04-23T14:36:11.951313+0000","last_undegraded":"2024-04-23T14:36:11.951313+0000","last_fullsized":"2024-04-23T14:36:11.951313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:11.951248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:11.951248+0000","last_clean_scrub_stamp":"2024-04-23T14:36:11.951248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:27:18.426162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":86,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:08.845073+0000","last_change":"2024-04-23T14:36:08.845094+0000","last_active":"2024-04-23T14:36:08.845073+0000","last_peered":"2024-04-23T14:36:08.845073+0000","last_clean":"2024-04-23T14:36:08.845073+0000","last_became_active":"2024-04-23T14:35:57.824039+0000","last_became_peered":"2024-04-23T14:35:57.824039+0000","last_unstale":"2024-04-23T14:36:08.845073+0000","last_undegraded":"2024-04-23T14:36:08.845073+0000","last_fullsized":"2024-04-23T14:36:08.845073+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:08.845012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:08.845012+0000","last_clean_scrub_stamp":"2024-04-23T14:36:08.845012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:48:11.993645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":99,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:09.226078+0000","last_change":"2024-04-23T14:36:09.226102+0000","last_active":"2024-04-23T14:36:09.226078+0000","last_peered":"2024-04-23T14:36:09.226078+0000","last_clean":"2024-04-23T14:36:09.226078+0000","last_became_active":"2024-04-23T14:35:57.696676+0000","last_became_peered":"2024-04-23T14:35:57.696676+0000","last_unstale":"2024-04-23T14:36:09.226078+0000","last_undegraded":"2024-04-23T14:36:09.226078+0000","last_fullsized":"2024-04-23T14:36:09.226078+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:09.226012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:09.226012+0000","last_clean_scrub_stamp":"2024-04-23T14:36:09.226012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:24:48.807013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":96,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:09.968572+0000","last_change":"2024-04-23T14:36:09.968627+0000","last_active":"2024-04-23T14:36:09.968572+0000","last_peered":"2024-04-23T14:36:09.968572+0000","last_clean":"2024-04-23T14:36:09.968572+0000","last_became_active":"2024-04-23T14:35:57.697166+0000","last_became_peered":"2024-04-23T14:35:57.697166+0000","last_unstale":"2024-04-23T14:36:09.968572+0000","last_undegraded":"2024-04-23T14:36:09.968572+0000","last_fullsized":"2024-04-23T14:36:09.968572+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:09.968437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:09.968437+0000","last_clean_scrub_stamp":"2024-04-23T14:36:09.968437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:59:37.190566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:11.769366+0000","last_change":"2024-04-23T14:36:11.769400+0000","last_active":"2024-04-23T14:36:11.769366+0000","last_peered":"2024-04-23T14:36:11.769366+0000","last_clean":"2024-04-23T14:36:11.769366+0000","last_became_active":"2024-04-23T14:35:46.624268+0000","last_became_peered":"2024-04-23T14:35:46.624268+0000","last_unstale":"2024-04-23T14:36:11.769366+0000","last_undegraded":"2024-04-23T14:36:11.769366+0000","last_fullsized":"2024-04-23T14:36:11.769366+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:11.769283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:11.769283+0000","last_clean_scrub_stamp":"2024-04-23T14:36:11.769283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:35.758841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":92,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-23T14:36:10.168712+0000","last_change":"2024-04-23T14:36:10.168731+0000","last_active":"2024-04-23T14:36:10.168712+0000","last_peered":"2024-04-23T14:36:10.168712+0000","last_clean":"2024-04-23T14:36:10.168712+0000","last_became_active":"2024-04-23T14:35:46.621458+0000","last_became_peered":"2024-04-23T14:35:46.621458+0000","last_unstale":"2024-04-23T14:36:10.168712+0000","last_undegraded":"2024-04-23T14:36:10.168712+0000","last_fullsized":"2024-04-23T14:36:10.168712+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:10.168654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:10.168654+0000","last_clean_scrub_stamp":"2024-04-23T14:36:10.168654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:29:23.395178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462941,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462941,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":57,"seq":244813135901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":40,"seq":171798691869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829912,"statfs":{"total":107374182400,"available":107345829888,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462941,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27680,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829920,"statfs":{"total":107374182400,"available":107345838080,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":59,"seq":253403070493,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829916,"statfs":{"total":107374182400,"available":107345833984,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":47,"seq":201863462942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829908,"statfs":{"total":107374182400,"available":107345825792,"internally_reserved":0,"allocated":679936,"data_stored":330863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:36:32.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-23T14:36:32.785 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-23T14:36:32.786 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-23T14:36:32.786 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-23T14:36:32.786 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:32.868 INFO:tasks.ceph.osd.0:Stopped 2024-04-23T14:36:32.869 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-23T14:36:32.869 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:32.946 INFO:tasks.ceph.osd.1:Stopped 2024-04-23T14:36:32.946 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-23T14:36:32.947 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:33.008 INFO:tasks.ceph.osd.2:Stopped 2024-04-23T14:36:33.008 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-23T14:36:33.008 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:33.103 INFO:tasks.ceph.osd.3:Stopped 2024-04-23T14:36:33.103 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-23T14:36:33.103 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:33.199 INFO:tasks.ceph.osd.4:Stopped 2024-04-23T14:36:33.199 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-23T14:36:33.199 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:33.285 INFO:tasks.ceph.osd.5:Stopped 2024-04-23T14:36:33.286 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-23T14:36:33.286 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:33.349 INFO:tasks.ceph.osd.6:Stopped 2024-04-23T14:36:33.350 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-23T14:36:33.350 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:33.417 INFO:tasks.ceph.osd.7:Stopped 2024-04-23T14:36:33.417 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-04-23T14:36:33.417 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:37.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2024-04-23T14:36:39.518 INFO:tasks.ceph.osd.8:Stopped 2024-04-23T14:36:39.519 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-04-23T14:36:39.519 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:39.607 INFO:tasks.ceph.osd.9:Stopped 2024-04-23T14:36:39.608 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-23T14:36:39.608 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-23T14:36:39.608 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:39.638 INFO:tasks.ceph.mgr.x:Stopped 2024-04-23T14:36:39.638 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-23T14:36:39.638 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-23T14:36:39.638 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T14:36:39.695 INFO:tasks.ceph.mon.a:Stopped 2024-04-23T14:36:39.695 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-23T14:36:39.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-23T14:36:39.749 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:36:39.749 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-23T14:36:40.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:36:40.045 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-23T14:36:40.242 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:36:40.242 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-23T14:36:40.452 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:36:40.452 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-23T14:36:40.633 INFO:tasks.ceph:Archiving mon data... 2024-04-23T14:36:40.634 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603/data/mon.a.tgz 2024-04-23T14:36:40.634 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-04-23T14:36:40.640 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.JtTM4fg9D8 2024-04-23T14:36:40.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.JtTM4fg9D8 2024-04-23T14:36:40.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.JtTM4fg9D8 2024-04-23T14:36:40.801 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.JtTM4fg9D8 is 209KB 2024-04-23T14:36:40.859 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.JtTM4fg9D8 2024-04-23T14:36:40.864 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-23T14:36:40.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-23T14:36:40.969 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-23T14:36:40.969 INFO:tasks.ceph:Archiving crash dumps... 2024-04-23T14:36:40.971 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603/remote/smithi084/crash 2024-04-23T14:36:40.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-23T14:36:41.025 INFO:tasks.ceph:Compressing logs... 2024-04-23T14:36:41.025 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T14:36:41.088 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37347.log 2024-04-23T14:36:41.088 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35284.log 2024-04-23T14:36:41.090 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35619.log 2024-04-23T14:36:41.090 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37347.log.gz 2024-04-23T14:36:41.090 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35395.log 2024-04-23T14:36:41.090 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35284.log.gz 2024-04-23T14:36:41.090 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35619.log.gz 2024-04-23T14:36:41.090 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36510.log 2024-04-23T14:36:41.093 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37952.log 2024-04-23T14:36:41.093 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35395.log: /var/log/ceph/ceph-client.admin.36510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37586.log 2024-04-23T14:36:41.093 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36510.log.gz 2024-04-23T14:36:41.093 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35395.log.gz 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37372.log 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37952.log.gz 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37586.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37586.log.gz --verbose 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-osd.0.log 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36634.log 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37372.log.gz 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36462.log 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36707.log 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36634.log.gz 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37978.log 2024-04-23T14:36:41.094 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36462.log.gz 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36707.log.gz 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37820.log 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37978.log.gz 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28571.log 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37820.log.gz 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36535.log 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38082.log 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28571.log.gz 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35605.log 2024-04-23T14:36:41.100 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36535.log.gz 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28455.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38082.log.gz 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37249.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35605.log.gz 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28455.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28455.log.gz 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.37249.log: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37249.log.gz 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37028.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36585.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37028.log.gz 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-23T14:36:41.101 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35886.log 2024-04-23T14:36:41.102 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36585.log.gz 2024-04-23T14:36:41.102 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36856.log 2024-04-23T14:36:41.108 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35886.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35886.log.gz 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36905.log 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36856.log.gz 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28383.log 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36560.log 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36905.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28383.log: 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28383.log.gz 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36560.log.gz 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37768.log 2024-04-23T14:36:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35374.log 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37768.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37768.log.gz 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37322.log 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35374.log.gz 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37690.log 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35397.log 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37690.log: /var/log/ceph/ceph-client.admin.37322.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37690.log.gz 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37322.log.gz 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38030.log 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37560.log 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35397.log: /var/log/ceph/ceph-client.admin.38030.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35397.log.gz 2024-04-23T14:36:41.110 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38030.log.gz 2024-04-23T14:36:41.111 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35383.log 2024-04-23T14:36:41.111 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36194.log 2024-04-23T14:36:41.111 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37560.log: /var/log/ceph/ceph-client.admin.35383.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37560.log.gz 2024-04-23T14:36:41.111 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35383.log.gz 2024-04-23T14:36:41.111 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35596.log 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35385.log 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36194.log: /var/log/ceph/ceph-client.admin.35596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36194.log.gz 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35596.log.gz 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37872.log 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28796.log 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35385.log: /var/log/ceph/ceph-client.admin.37872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35385.log.gz 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37872.log.gz 2024-04-23T14:36:41.114 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36980.log 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36438.log 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28796.log: /var/log/ceph/ceph-client.admin.36980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28796.log.gz 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36980.log.gz 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36955.log 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35634.log 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36438.log: /var/log/ceph/ceph-client.admin.36955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36438.log.gz 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36955.log.gz 2024-04-23T14:36:41.115 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37846.log 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37152.log 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35634.log: /var/log/ceph/ceph-client.admin.37846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35634.log.gz 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37846.log.gz 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35398.log 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36756.log 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37152.log: /var/log/ceph/ceph-client.admin.35398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37152.log.gz 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35398.log.gz 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35647.log 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36659.log 2024-04-23T14:36:41.118 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36756.log: /var/log/ceph/ceph-client.admin.35647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36756.log.gz 2024-04-23T14:36:41.119 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35647.log.gz 2024-04-23T14:36:41.119 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36310.log 2024-04-23T14:36:41.121 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35650.log 2024-04-23T14:36:41.121 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36659.log: /var/log/ceph/ceph-client.admin.36310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36659.log.gz 2024-04-23T14:36:41.121 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36310.log.gz 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38212.log 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28218.log 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35650.log: /var/log/ceph/ceph-client.admin.38212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35650.log.gz 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38212.log.gz 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36486.log 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28218.log: /var/log/ceph/ceph-client.admin.36486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28218.log.gz 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36486.log.gz 2024-04-23T14:36:41.122 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37926.log 2024-04-23T14:36:41.123 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36362.log 2024-04-23T14:36:41.123 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.37926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37926.log.gz 2024-04-23T14:36:41.128 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-23T14:36:41.129 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36362.log.gz 2024-04-23T14:36:41.130 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35662.log 2024-04-23T14:36:41.145 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35640.log 2024-04-23T14:36:41.145 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35662.log.gz 2024-04-23T14:36:41.153 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35207.log 2024-04-23T14:36:41.153 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35640.log.gz 2024-04-23T14:36:41.156 INFO:teuthology.orchestra.run.smithi084.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-23T14:36:41.156 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28431.log 2024-04-23T14:36:41.157 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35207.log.gz 2024-04-23T14:36:41.157 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37898.log 2024-04-23T14:36:41.158 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28431.log.gz 2024-04-23T14:36:41.158 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28335.log 2024-04-23T14:36:41.159 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37898.log: 92.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37898.log.gz 2024-04-23T14:36:41.159 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-04-23T14:36:41.159 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35386.log 2024-04-23T14:36:41.163 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37447.log 2024-04-23T14:36:41.163 INFO:teuthology.orchestra.run.smithi084.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-04-23T14:36:41.163 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28335.log.gz 2024-04-23T14:36:41.163 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35386.log.gz 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37447.log.gz 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35310.log 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.37473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35936.log 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35310.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35310.log.gz 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35936.log.gz 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36806.log 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35639.log 2024-04-23T14:36:41.164 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37297.log 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36806.log.gz 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35639.log.gz 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28311.log 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36116.log 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37297.log.gz 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35905.log 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28311.log.gz 2024-04-23T14:36:41.171 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36116.log.gz 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37742.log 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37612.log 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35905.log.gz 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37128.log 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37742.log.gz 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37612.log.gz 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38160.log 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37225.log 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37128.log.gz 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28359.log 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38160.log.gz 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37225.log.gz 2024-04-23T14:36:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-23T14:36:41.173 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36610.log 2024-04-23T14:36:41.173 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28359.log.gz 2024-04-23T14:36:41.173 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37177.log 2024-04-23T14:36:41.173 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36610.log.gz 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37177.log.gz 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35903.log 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28287.log 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35903.log.gz 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-23T14:36:41.178 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38186.log 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37004.log 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28287.log.gz 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38186.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20253.log 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38186.log.gz 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37004.log.gz 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28152.log 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.tmp-client.admin.20253.log: /var/log/ceph/ceph-client.admin.38288.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20253.log.gz 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36831.log 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28479.log 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28152.log: /var/log/ceph/ceph-client.admin.36831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28152.log.gz 2024-04-23T14:36:41.179 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36831.log.gz 2024-04-23T14:36:41.180 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37273.log 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36731.log 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28479.log.gz 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37273.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28510.log 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37273.log.gz 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37794.log 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0%/var/log/ceph/ceph-client.admin.28510.log: -- replaced with /var/log/ceph/ceph-client.admin.36731.log.gz 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28510.log.gz 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37078.log 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35258.log 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37794.log.gz 2024-04-23T14:36:41.191 INFO:teuthology.orchestra.run.smithi084.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35906.log 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37078.log: /var/log/ceph/ceph-client.admin.35258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37078.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35920.log 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35258.log.gz 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35906.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35906.log.gz 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35920.log.gz 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36142.log 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37397.log 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.36142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36142.log.gz 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37053.log 2024-04-23T14:36:41.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38004.log 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37397.log.gz 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37053.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36930.log 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37053.log.gz 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38004.log.gz 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38264.log 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35882.log 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36930.log: /var/log/ceph/ceph-client.admin.38264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36930.log.gz 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38264.log.gz 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35393.log 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37103.log 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35882.log.gz 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35393.log.gz 2024-04-23T14:36:41.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37103.log.gz 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35697.log 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37534.log 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35902.log 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35697.log.gz 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37534.log.gz 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36683.log 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38238.log 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35902.log.gz 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36683.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-23T14:36:41.194 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28407.log 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38238.log.gz 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35336.log 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.28407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28407.log.gz 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37508.log 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35336.log.gz 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36781.log 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37508.log.gz 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38056.log 2024-04-23T14:36:41.201 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36781.log.gz 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36338.log 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38056.log.gz 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37201.log 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36338.log.gz 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36414.log 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37201.log: 92.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37201.log.gz 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35373.log 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38134.log 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36414.log.gz 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35373.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35232.log 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35373.log.gz 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38134.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36168.log 2024-04-23T14:36:41.202 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-04-23T14:36:41.203 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35904.log 2024-04-23T14:36:41.203 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35232.log.gz 2024-04-23T14:36:41.204 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.36168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36881.log 2024-04-23T14:36:41.204 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36168.log.gz 2024-04-23T14:36:41.204 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38108.log 2024-04-23T14:36:41.204 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35904.log: /var/log/ceph/ceph-client.admin.36881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35904.log.gz 2024-04-23T14:36:41.204 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36881.log.gz 2024-04-23T14:36:41.204 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38108.log.gz 2024-04-23T14:36:41.205 INFO:teuthology.orchestra.run.smithi084.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-23T14:36:41.205 INFO:teuthology.orchestra.run.smithi084.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-23T14:36:41.230 INFO:teuthology.orchestra.run.smithi084.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-23T14:36:41.243 INFO:teuthology.orchestra.run.smithi084.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-04-23T14:36:41.269 INFO:teuthology.orchestra.run.smithi084.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-23T14:36:41.272 INFO:teuthology.orchestra.run.smithi084.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-23T14:36:41.273 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-04-23T14:36:41.274 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m0.202s 2024-04-23T14:36:41.274 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m0.966s 2024-04-23T14:36:41.274 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.094s 2024-04-23T14:36:41.274 INFO:tasks.ceph:Archiving logs... 2024-04-23T14:36:41.274 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603/remote/smithi084/log 2024-04-23T14:36:41.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T14:36:41.523 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T14:36:41.540 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T14:36:41.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T14:36:41.578 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T14:36:41.586 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T14:36:41.586 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:36:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: remote refid st t when poll reach delay offset jitter 2024-04-23T14:36:41.607 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================== 2024-04-23T14:36:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: hv01.front.sepi 44.4.53.2 2 u 23 64 37 0.094 -0.565 0.575 2024-04-23T14:36:41.607 INFO:teuthology.orchestra.run.smithi084.stdout:+hv02.front.sepi 63.231.80.2 3 u 23 64 37 0.128 +0.739 0.480 2024-04-23T14:36:41.607 INFO:teuthology.orchestra.run.smithi084.stdout:*hv03.front.sepi 74.6.168.72 3 u 23 64 37 0.131 -0.804 0.606 2024-04-23T14:36:41.607 INFO:teuthology.orchestra.run.smithi084.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-23T14:36:41.608 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T14:36:41.616 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T14:36:41.617 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T14:36:41.627 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T14:36:41.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T14:36:41.645 INFO:teuthology.task.internal:Duration was 726.478120 seconds 2024-04-23T14:36:41.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T14:36:41.654 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T14:36:41.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T14:36:41.692 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T14:36:41.692 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-04-23T14:36:41.692 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T14:36:41.742 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T14:36:41.742 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T14:36:41.810 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T14:36:41.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T14:36:41.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T14:36:41.917 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T14:36:41.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T14:36:41.928 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T14:36:41.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T14:36:41.981 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-04-23T14:36:41.995 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T14:36:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:36:42.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T14:36:42.051 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T14:36:42.051 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-23_14:14:08-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7669603/remote/smithi084 2024-04-23T14:36:42.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T14:36:42.104 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T14:36:42.104 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T14:36:42.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T14:36:42.156 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T14:36:42.156 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T14:36:42.164 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T14:36:42.165 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T14:36:42.193 INFO:teuthology.orchestra.run.smithi084.stdout: 658497 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 14:36 /home/ubuntu/cephtest 2024-04-23T14:36:42.195 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T14:36:42.204 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T14:36:42.226 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} duration: 726.4781200885773 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-23T14:36:42.226 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T14:36:42.324 INFO:teuthology.run:pass