2026-02-22T22:55:05.134 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T22:55:05.151 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T22:55:05.228 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '64311' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - overall HEALTH_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5907 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+pqmRDeK/i+ZrJqFYf4dHmFkLOFO7XJRfTTRa4EIPYeI6K11h1L+r4wDMbslfvUPqBS1cmhnTN9r6IvawHoQQ= tasks: - install: null - ceph: log-ignorelist: - \(POOL_APP_NOT_ENABLED\) - workunit: clients: all: - cephtool/test_kvstore_tool.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:40 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T22:55:05.229 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T22:55:05.229 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T22:55:05.229 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T22:55:05.231 INFO:teuthology.task.internal:Checking packages... 2026-02-22T22:55:05.231 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T22:55:05.231 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T22:55:05.231 INFO:teuthology.packaging:ref: None 2026-02-22T22:55:05.231 INFO:teuthology.packaging:tag: None 2026-02-22T22:55:05.231 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:55:05.231 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:55:05.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:55:05.364 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:55:05.366 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T22:55:05.376 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T22:55:05.377 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T22:55:05.390 INFO:teuthology.task.internal:Saving configuration 2026-02-22T22:55:05.399 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T22:55:05.410 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T22:55:05.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial045.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 22:51:27.083894', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGOCbCq9vOyl0TADSjuE2JDbGt9TNxdDhqTkKefgiZW4'} 2026-02-22T22:55:05.489 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T22:55:05.500 INFO:teuthology.task.internal:roles: ubuntu@trial045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-22T22:55:05.500 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T22:55:05.599 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x788e2b10cb80>, signals=[15]) 2026-02-22T22:55:05.600 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T22:55:05.611 INFO:teuthology.task.internal:Opening connections... 2026-02-22T22:55:05.611 DEBUG:teuthology.task.internal:connecting to ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:55:05.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T22:55:05.674 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T22:55:05.685 DEBUG:teuthology.orchestra.run.trial045:> uname -m 2026-02-22T22:55:05.751 INFO:teuthology.orchestra.run.trial045.stdout:x86_64 2026-02-22T22:55:05.752 DEBUG:teuthology.orchestra.run.trial045:> cat /etc/os-release 2026-02-22T22:55:05.797 INFO:teuthology.orchestra.run.trial045.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:NAME="Ubuntu" 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:VERSION_ID="22.04" 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:VERSION_CODENAME=jammy 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:ID=ubuntu 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:ID_LIKE=debian 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T22:55:05.798 INFO:teuthology.orchestra.run.trial045.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T22:55:05.799 INFO:teuthology.orchestra.run.trial045.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T22:55:05.799 INFO:teuthology.orchestra.run.trial045.stdout:UBUNTU_CODENAME=jammy 2026-02-22T22:55:05.799 INFO:teuthology.lock.ops:Updating trial045.front.sepia.ceph.com on lock server 2026-02-22T22:55:05.887 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T22:55:05.899 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T22:55:05.911 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T22:55:05.911 DEBUG:teuthology.orchestra.run.trial045:> test '!' -e /home/ubuntu/cephtest 2026-02-22T22:55:05.914 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T22:55:05.924 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T22:55:05.924 DEBUG:teuthology.orchestra.run.trial045:> test -z $(ls -A /var/lib/ceph) 2026-02-22T22:55:05.958 INFO:teuthology.orchestra.run.trial045.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T22:55:05.959 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T22:55:06.057 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T22:55:06.077 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T22:55:06.077 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T22:55:06.077 DEBUG:teuthology.orchestra.run.trial045:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T22:55:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:55:06.081 DEBUG:teuthology.orchestra.run.trial045:> uname -r 2026-02-22T22:55:06.125 INFO:teuthology.orchestra.run.trial045.stdout:5.15.0-170-generic 2026-02-22T22:55:06.125 INFO:teuthology.task.kernel:Running kernel on trial045: 5.15.0-170-generic 2026-02-22T22:55:06.125 DEBUG:teuthology.orchestra.run.trial045:> sudo apt-get clean 2026-02-22T22:55:06.184 DEBUG:teuthology.orchestra.run.trial045:> sudo apt-get update 2026-02-22T22:55:06.377 INFO:teuthology.orchestra.run.trial045.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T22:55:06.384 INFO:teuthology.orchestra.run.trial045.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T22:55:06.424 INFO:teuthology.orchestra.run.trial045.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T22:55:06.439 INFO:teuthology.orchestra.run.trial045.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T22:55:06.517 INFO:teuthology.orchestra.run.trial045.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T22:55:06.583 INFO:teuthology.orchestra.run.trial045.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T22:55:06.584 INFO:teuthology.orchestra.run.trial045.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T22:55:06.585 INFO:teuthology.orchestra.run.trial045.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T22:55:06.591 INFO:teuthology.orchestra.run.trial045.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T22:55:06.593 INFO:teuthology.orchestra.run.trial045.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T22:55:06.593 INFO:teuthology.orchestra.run.trial045.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T22:55:06.619 INFO:teuthology.orchestra.run.trial045.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T22:55:06.623 INFO:teuthology.orchestra.run.trial045.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T22:55:06.634 INFO:teuthology.orchestra.run.trial045.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T22:55:06.635 INFO:teuthology.orchestra.run.trial045.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T22:55:06.635 INFO:teuthology.orchestra.run.trial045.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T22:55:06.639 INFO:teuthology.orchestra.run.trial045.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T22:55:06.639 INFO:teuthology.orchestra.run.trial045.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T22:55:06.657 INFO:teuthology.orchestra.run.trial045.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T22:55:10.565 INFO:teuthology.orchestra.run.trial045.stdout:Fetched 22.6 MB in 2s (13.4 MB/s) 2026-02-22T22:55:11.103 INFO:teuthology.orchestra.run.trial045.stdout:Reading package lists... 2026-02-22T22:55:11.117 DEBUG:teuthology.orchestra.run.trial045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T22:55:11.150 INFO:teuthology.orchestra.run.trial045.stdout:Reading package lists... 2026-02-22T22:55:11.259 INFO:teuthology.orchestra.run.trial045.stdout:Building dependency tree... 2026-02-22T22:55:11.259 INFO:teuthology.orchestra.run.trial045.stdout:Reading state information... 2026-02-22T22:55:11.346 INFO:teuthology.orchestra.run.trial045.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T22:55:11.347 INFO:teuthology.orchestra.run.trial045.stdout:linux-image-generic set to manually installed. 2026-02-22T22:55:11.347 INFO:teuthology.orchestra.run.trial045.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T22:55:11.347 INFO:teuthology.orchestra.run.trial045.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T22:55:11.347 INFO:teuthology.orchestra.run.trial045.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T22:55:11.347 INFO:teuthology.orchestra.run.trial045.stdout: sg3-utils-udev 2026-02-22T22:55:11.347 INFO:teuthology.orchestra.run.trial045.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T22:55:11.366 INFO:teuthology.orchestra.run.trial045.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T22:55:11.368 DEBUG:teuthology.orchestra.run.trial045:> dpkg -s linux-image-generic 2026-02-22T22:55:11.424 INFO:teuthology.orchestra.run.trial045.stdout:Package: linux-image-generic 2026-02-22T22:55:11.424 INFO:teuthology.orchestra.run.trial045.stdout:Status: install ok installed 2026-02-22T22:55:11.424 INFO:teuthology.orchestra.run.trial045.stdout:Priority: optional 2026-02-22T22:55:11.424 INFO:teuthology.orchestra.run.trial045.stdout:Section: kernel 2026-02-22T22:55:11.424 INFO:teuthology.orchestra.run.trial045.stdout:Installed-Size: 21 2026-02-22T22:55:11.424 INFO:teuthology.orchestra.run.trial045.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout:Architecture: amd64 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout:Source: linux-meta 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout:Version: 5.15.0.170.159 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout:Recommends: thermald 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout:Description: Generic Linux kernel image 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout: This package will always depend on the latest generic kernel image 2026-02-22T22:55:11.425 INFO:teuthology.orchestra.run.trial045.stdout: available. 2026-02-22T22:55:11.426 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T22:55:11.426 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-22T22:55:11.426 DEBUG:teuthology.orchestra.run.trial045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T22:55:11.474 INFO:teuthology.orchestra.run.trial045.stdout:ttyS1 2026-02-22T22:55:11.500 DEBUG:teuthology.parallel:result is None 2026-02-22T22:55:11.500 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T22:55:11.512 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T22:55:11.512 DEBUG:teuthology.orchestra.run.trial045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T22:55:11.546 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T22:55:11.558 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T22:55:11.572 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T22:55:11.572 DEBUG:teuthology.orchestra.run.trial045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T22:55:11.591 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T22:55:11.602 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T22:55:11.603 DEBUG:teuthology.orchestra.run.trial045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T22:55:11.639 INFO:teuthology.orchestra.run.trial045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:11.645 INFO:teuthology.orchestra.run.trial045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:11.647 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T22:55:11.658 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T22:55:11.658 DEBUG:teuthology.orchestra.run.trial045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T22:55:11.698 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T22:55:11.714 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T22:55:11.715 DEBUG:teuthology.orchestra.run.trial045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T22:55:11.746 DEBUG:teuthology.orchestra.run.trial045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T22:55:11.790 DEBUG:teuthology.orchestra.run.trial045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T22:55:11.834 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:55:11.834 DEBUG:teuthology.orchestra.run.trial045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T22:55:11.883 DEBUG:teuthology.orchestra.run.trial045:> sudo service rsyslog restart 2026-02-22T22:55:11.942 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T22:55:11.955 INFO:teuthology.task.internal:Starting timer... 2026-02-22T22:55:11.955 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T22:55:11.976 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T22:55:11.994 DEBUG:teuthology.task.selinux:Excluding trial045: OS 'ubuntu' does not support SELinux 2026-02-22T22:55:11.994 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T22:55:11.994 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-22T22:55:11.994 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T22:55:11.995 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T22:55:12.010 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T22:55:12.022 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T22:55:12.024 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial045.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T22:57:38.089 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial045.front.sepia.ceph.com')] 2026-02-22T22:57:38.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial045.front.sepia.ceph.com' 2026-02-22T22:57:38.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T22:57:38.200 DEBUG:teuthology.orchestra.run.trial045:> true 2026-02-22T22:57:38.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial045.front.sepia.ceph.com' 2026-02-22T22:57:38.445 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T22:57:38.462 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T22:57:38.462 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T22:57:38.463 DEBUG:teuthology.orchestra.run.trial045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Command line: ntpd -gq 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: ---------------------------------------------------- 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: corporation. Support and training for ntp-4 are 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: available at https://www.nwtime.org/support 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: ---------------------------------------------------- 2026-02-22T22:57:38.506 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: proto: precision = 0.030 usec (-25) 2026-02-22T22:57:38.507 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: basedate set to 2022-02-04 2026-02-22T22:57:38.507 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: gps base set to 2022-02-06 (week 2196) 2026-02-22T22:57:38.507 INFO:teuthology.orchestra.run.trial045.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T22:57:38.507 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T22:57:38.507 INFO:teuthology.orchestra.run.trial045.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T22:57:38.507 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T22:57:38.508 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T22:57:38.508 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T22:57:38.508 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T22:57:38.508 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Listen normally on 3 enp1s0f0 10.20.193.45:123 2026-02-22T22:57:38.508 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Listen normally on 4 lo [::1]:123 2026-02-22T22:57:38.508 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:5f04%2]:123 2026-02-22T22:57:38.508 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:37 ntpd[9418]: Listening on routing socket on fd #22 for interface updates 2026-02-22T22:57:39.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:39.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:39.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:39.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:39.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:39.574 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:39.574 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:39.574 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:39.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:39.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:39.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:39.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:38 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:41.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:41.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:41.508 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:41.508 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:41.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:41.574 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:41.574 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:41.574 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:41.599 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:41.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:41.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:41.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:40 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:42.506 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:41 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:42.506 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:41 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:42.530 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:41 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:42.531 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:41 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:42.531 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:41 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:42.531 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:41 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:43.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:43.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:43.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:43.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:43.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:43.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:43.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:43.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:43.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:43.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:43.576 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:43.576 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:43.576 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:43.576 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:43.599 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:43.599 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:43.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:43.600 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:42 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:44.506 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:43 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:44.506 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:43 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:44.530 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:43 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:44.530 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:43 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:44.530 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:43 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:44.530 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:43 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:45.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:45.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:45.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:45.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:45.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:45.507 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:45.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T22:57:45.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T22:57:45.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T22:57:45.573 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:44 ntpd[9418]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T22:57:45.574 INFO:teuthology.orchestra.run.trial045.stdout:22 Feb 22:57:45 ntpd[9418]: ntpd: time set +0.775177 s 2026-02-22T22:57:45.574 INFO:teuthology.orchestra.run.trial045.stdout:ntpd: time set +0.775177s 2026-02-22T22:57:45.575 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:45 ntpd[9418]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T22:57:45.575 INFO:teuthology.orchestra.run.trial045.stderr:22 Feb 22:57:45 ntpd[9418]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T22:57:45.612 INFO:teuthology.orchestra.run.trial045.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T22:57:45.612 INFO:teuthology.orchestra.run.trial045.stdout:============================================================================== 2026-02-22T22:57:45.613 INFO:teuthology.orchestra.run.trial045.stdout: LAX.CALTICK.NET .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:57:45.613 INFO:teuthology.orchestra.run.trial045.stdout: 139-177-202-26. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:57:45.613 INFO:teuthology.orchestra.run.trial045.stdout: 45.83.234.123 ( .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:57:45.613 INFO:teuthology.orchestra.run.trial045.stdout: 44.190.5.123 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T22:57:45.614 INFO:teuthology.run_tasks:Running task install... 2026-02-22T22:57:45.630 DEBUG:teuthology.task.install:project ceph 2026-02-22T22:57:45.630 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T22:57:45.630 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T22:57:45.630 INFO:teuthology.task.install:Using flavor: default 2026-02-22T22:57:45.640 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T22:57:45.640 INFO:teuthology.task.install:extra packages: [] 2026-02-22T22:57:45.641 DEBUG:teuthology.orchestra.run.trial045:> sudo apt-key list | grep Ceph 2026-02-22T22:57:45.704 INFO:teuthology.orchestra.run.trial045.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T22:57:45.734 INFO:teuthology.orchestra.run.trial045.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T22:57:45.734 INFO:teuthology.orchestra.run.trial045.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T22:57:45.735 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-22T22:57:45.735 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:57:45.909 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T22:57:45.909 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:57:46.048 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:57:46.048 DEBUG:teuthology.orchestra.run.trial045:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T22:57:46.058 DEBUG:teuthology.orchestra.run.trial045:> sudo apt-get update 2026-02-22T22:57:46.223 INFO:teuthology.orchestra.run.trial045.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T22:57:46.234 INFO:teuthology.orchestra.run.trial045.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T22:57:46.246 INFO:teuthology.orchestra.run.trial045.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T22:57:46.258 INFO:teuthology.orchestra.run.trial045.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T22:57:46.568 INFO:teuthology.orchestra.run.trial045.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T22:57:46.649 INFO:teuthology.orchestra.run.trial045.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T22:57:46.737 INFO:teuthology.orchestra.run.trial045.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T22:57:46.825 INFO:teuthology.orchestra.run.trial045.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T22:57:46.968 INFO:teuthology.orchestra.run.trial045.stdout:Fetched 25.8 kB in 1s (34.3 kB/s) 2026-02-22T22:57:47.495 INFO:teuthology.orchestra.run.trial045.stdout:Reading package lists... 2026-02-22T22:57:47.508 DEBUG:teuthology.orchestra.run.trial045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:57:47.542 INFO:teuthology.orchestra.run.trial045.stdout:Reading package lists... 2026-02-22T22:57:47.650 INFO:teuthology.orchestra.run.trial045.stdout:Building dependency tree... 2026-02-22T22:57:47.651 INFO:teuthology.orchestra.run.trial045.stdout:Reading state information... 2026-02-22T22:57:47.743 INFO:teuthology.orchestra.run.trial045.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T22:57:47.743 INFO:teuthology.orchestra.run.trial045.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout: sg3-utils-udev 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout:The following additional packages will be installed: 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T22:57:47.744 INFO:teuthology.orchestra.run.trial045.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T22:57:47.745 INFO:teuthology.orchestra.run.trial045.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T22:57:47.745 INFO:teuthology.orchestra.run.trial045.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T22:57:47.745 INFO:teuthology.orchestra.run.trial045.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T22:57:47.745 INFO:teuthology.orchestra.run.trial045.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T22:57:47.745 INFO:teuthology.orchestra.run.trial045.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T22:57:47.745 INFO:teuthology.orchestra.run.trial045.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T22:57:47.745 INFO:teuthology.orchestra.run.trial045.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T22:57:47.746 INFO:teuthology.orchestra.run.trial045.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout:Suggested packages: 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout:Recommended packages: 2026-02-22T22:57:47.747 INFO:teuthology.orchestra.run.trial045.stdout: btrfs-tools 2026-02-22T22:57:47.778 INFO:teuthology.orchestra.run.trial045.stdout:The following NEW packages will be installed: 2026-02-22T22:57:47.778 INFO:teuthology.orchestra.run.trial045.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T22:57:47.778 INFO:teuthology.orchestra.run.trial045.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T22:57:47.778 INFO:teuthology.orchestra.run.trial045.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T22:57:47.778 INFO:teuthology.orchestra.run.trial045.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T22:57:47.778 INFO:teuthology.orchestra.run.trial045.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T22:57:47.778 INFO:teuthology.orchestra.run.trial045.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T22:57:47.779 INFO:teuthology.orchestra.run.trial045.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T22:57:47.780 INFO:teuthology.orchestra.run.trial045.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T22:57:47.780 INFO:teuthology.orchestra.run.trial045.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T22:57:47.780 INFO:teuthology.orchestra.run.trial045.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T22:57:47.780 INFO:teuthology.orchestra.run.trial045.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2026-02-22T22:57:47.780 INFO:teuthology.orchestra.run.trial045.stdout: xmlstarlet zip 2026-02-22T22:57:47.857 INFO:teuthology.orchestra.run.trial045.stdout:0 upgraded, 108 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T22:57:47.857 INFO:teuthology.orchestra.run.trial045.stdout:Need to get 178 MB of archives. 2026-02-22T22:57:47.857 INFO:teuthology.orchestra.run.trial045.stdout:After this operation, 809 MB of additional disk space will be used. 2026-02-22T22:57:47.857 INFO:teuthology.orchestra.run.trial045.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T22:57:47.959 INFO:teuthology.orchestra.run.trial045.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T22:57:48.003 INFO:teuthology.orchestra.run.trial045.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T22:57:48.019 INFO:teuthology.orchestra.run.trial045.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T22:57:48.025 INFO:teuthology.orchestra.run.trial045.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T22:57:48.025 INFO:teuthology.orchestra.run.trial045.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T22:57:48.028 INFO:teuthology.orchestra.run.trial045.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T22:57:48.029 INFO:teuthology.orchestra.run.trial045.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T22:57:48.035 INFO:teuthology.orchestra.run.trial045.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T22:57:48.132 INFO:teuthology.orchestra.run.trial045.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T22:57:48.150 INFO:teuthology.orchestra.run.trial045.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T22:57:48.151 INFO:teuthology.orchestra.run.trial045.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T22:57:48.157 INFO:teuthology.orchestra.run.trial045.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T22:57:48.467 INFO:teuthology.orchestra.run.trial045.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T22:57:48.478 INFO:teuthology.orchestra.run.trial045.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T22:57:48.547 INFO:teuthology.orchestra.run.trial045.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T22:57:48.596 INFO:teuthology.orchestra.run.trial045.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T22:57:48.677 INFO:teuthology.orchestra.run.trial045.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T22:57:48.680 INFO:teuthology.orchestra.run.trial045.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T22:57:48.683 INFO:teuthology.orchestra.run.trial045.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T22:57:48.700 INFO:teuthology.orchestra.run.trial045.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T22:57:48.748 INFO:teuthology.orchestra.run.trial045.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T22:57:48.759 INFO:teuthology.orchestra.run.trial045.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T22:57:48.785 INFO:teuthology.orchestra.run.trial045.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T22:57:48.793 INFO:teuthology.orchestra.run.trial045.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T22:57:48.795 INFO:teuthology.orchestra.run.trial045.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T22:57:48.796 INFO:teuthology.orchestra.run.trial045.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T22:57:48.797 INFO:teuthology.orchestra.run.trial045.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T22:57:48.803 INFO:teuthology.orchestra.run.trial045.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T22:57:48.813 INFO:teuthology.orchestra.run.trial045.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T22:57:48.817 INFO:teuthology.orchestra.run.trial045.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T22:57:48.821 INFO:teuthology.orchestra.run.trial045.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T22:57:48.889 INFO:teuthology.orchestra.run.trial045.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T22:57:48.890 INFO:teuthology.orchestra.run.trial045.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T22:57:48.891 INFO:teuthology.orchestra.run.trial045.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T22:57:48.893 INFO:teuthology.orchestra.run.trial045.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T22:57:48.895 INFO:teuthology.orchestra.run.trial045.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T22:57:48.895 INFO:teuthology.orchestra.run.trial045.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T22:57:48.896 INFO:teuthology.orchestra.run.trial045.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T22:57:48.896 INFO:teuthology.orchestra.run.trial045.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T22:57:48.967 INFO:teuthology.orchestra.run.trial045.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T22:57:48.967 INFO:teuthology.orchestra.run.trial045.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T22:57:48.967 INFO:teuthology.orchestra.run.trial045.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T22:57:48.967 INFO:teuthology.orchestra.run.trial045.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T22:57:48.967 INFO:teuthology.orchestra.run.trial045.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T22:57:49.039 INFO:teuthology.orchestra.run.trial045.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T22:57:49.041 INFO:teuthology.orchestra.run.trial045.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T22:57:49.041 INFO:teuthology.orchestra.run.trial045.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T22:57:49.042 INFO:teuthology.orchestra.run.trial045.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T22:57:49.042 INFO:teuthology.orchestra.run.trial045.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T22:57:49.111 INFO:teuthology.orchestra.run.trial045.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T22:57:49.111 INFO:teuthology.orchestra.run.trial045.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T22:57:49.112 INFO:teuthology.orchestra.run.trial045.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T22:57:49.112 INFO:teuthology.orchestra.run.trial045.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T22:57:49.113 INFO:teuthology.orchestra.run.trial045.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T22:57:49.184 INFO:teuthology.orchestra.run.trial045.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T22:57:49.185 INFO:teuthology.orchestra.run.trial045.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T22:57:49.185 INFO:teuthology.orchestra.run.trial045.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T22:57:49.185 INFO:teuthology.orchestra.run.trial045.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T22:57:49.185 INFO:teuthology.orchestra.run.trial045.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T22:57:49.185 INFO:teuthology.orchestra.run.trial045.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T22:57:49.186 INFO:teuthology.orchestra.run.trial045.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T22:57:49.200 INFO:teuthology.orchestra.run.trial045.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T22:57:49.240 INFO:teuthology.orchestra.run.trial045.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T22:57:49.253 INFO:teuthology.orchestra.run.trial045.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T22:57:49.256 INFO:teuthology.orchestra.run.trial045.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T22:57:49.258 INFO:teuthology.orchestra.run.trial045.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T22:57:49.259 INFO:teuthology.orchestra.run.trial045.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T22:57:49.261 INFO:teuthology.orchestra.run.trial045.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T22:57:49.261 INFO:teuthology.orchestra.run.trial045.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T22:57:49.262 INFO:teuthology.orchestra.run.trial045.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T22:57:49.296 INFO:teuthology.orchestra.run.trial045.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T22:57:49.330 INFO:teuthology.orchestra.run.trial045.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T22:57:49.332 INFO:teuthology.orchestra.run.trial045.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T22:57:49.332 INFO:teuthology.orchestra.run.trial045.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T22:57:49.352 INFO:teuthology.orchestra.run.trial045.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T22:57:49.352 INFO:teuthology.orchestra.run.trial045.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T22:57:49.403 INFO:teuthology.orchestra.run.trial045.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T22:57:49.403 INFO:teuthology.orchestra.run.trial045.stdout:Get:79 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T22:57:49.403 INFO:teuthology.orchestra.run.trial045.stdout:Get:80 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T22:57:49.403 INFO:teuthology.orchestra.run.trial045.stdout:Get:81 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T22:57:49.407 INFO:teuthology.orchestra.run.trial045.stdout:Get:82 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T22:57:49.437 INFO:teuthology.orchestra.run.trial045.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T22:57:49.437 INFO:teuthology.orchestra.run.trial045.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T22:57:49.475 INFO:teuthology.orchestra.run.trial045.stdout:Get:85 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T22:57:49.476 INFO:teuthology.orchestra.run.trial045.stdout:Get:86 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T22:57:49.476 INFO:teuthology.orchestra.run.trial045.stdout:Get:87 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T22:57:49.479 INFO:teuthology.orchestra.run.trial045.stdout:Get:88 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T22:57:49.482 INFO:teuthology.orchestra.run.trial045.stdout:Get:89 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T22:57:49.546 INFO:teuthology.orchestra.run.trial045.stdout:Get:90 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T22:57:49.547 INFO:teuthology.orchestra.run.trial045.stdout:Get:91 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T22:57:49.551 INFO:teuthology.orchestra.run.trial045.stdout:Get:92 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T22:57:49.552 INFO:teuthology.orchestra.run.trial045.stdout:Get:93 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T22:57:49.555 INFO:teuthology.orchestra.run.trial045.stdout:Get:94 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T22:57:49.618 INFO:teuthology.orchestra.run.trial045.stdout:Get:95 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T22:57:49.619 INFO:teuthology.orchestra.run.trial045.stdout:Get:96 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T22:57:49.619 INFO:teuthology.orchestra.run.trial045.stdout:Get:97 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T22:57:49.620 INFO:teuthology.orchestra.run.trial045.stdout:Get:98 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T22:57:49.621 INFO:teuthology.orchestra.run.trial045.stdout:Get:99 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T22:57:49.690 INFO:teuthology.orchestra.run.trial045.stdout:Get:100 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T22:57:49.690 INFO:teuthology.orchestra.run.trial045.stdout:Get:101 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T22:57:49.691 INFO:teuthology.orchestra.run.trial045.stdout:Get:102 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T22:57:49.693 INFO:teuthology.orchestra.run.trial045.stdout:Get:103 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T22:57:49.693 INFO:teuthology.orchestra.run.trial045.stdout:Get:104 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T22:57:50.299 INFO:teuthology.orchestra.run.trial045.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T22:57:50.300 INFO:teuthology.orchestra.run.trial045.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T22:57:50.300 INFO:teuthology.orchestra.run.trial045.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T22:57:50.520 INFO:teuthology.orchestra.run.trial045.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T22:57:50.849 INFO:teuthology.orchestra.run.trial045.stdout:Fetched 178 MB in 3s (64.8 MB/s) 2026-02-22T22:57:50.869 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T22:57:51.030 INFO:teuthology.orchestra.run.trial045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T22:57:51.032 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T22:57:51.034 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T22:57:51.055 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T22:57:51.067 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T22:57:51.068 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T22:57:51.087 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T22:57:51.099 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T22:57:51.099 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T22:57:51.118 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T22:57:51.130 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T22:57:51.135 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:57:51.172 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T22:57:51.181 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T22:57:51.181 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:57:51.197 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T22:57:51.205 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T22:57:51.205 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:57:51.225 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T22:57:51.234 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T22:57:51.234 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T22:57:51.251 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package librados2. 2026-02-22T22:57:51.258 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.259 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.301 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libnbd0. 2026-02-22T22:57:51.310 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T22:57:51.310 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T22:57:51.325 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package librbd1. 2026-02-22T22:57:51.333 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.333 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.375 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T22:57:51.383 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.384 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.405 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-rados. 2026-02-22T22:57:51.413 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.414 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.430 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T22:57:51.440 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:51.440 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.454 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T22:57:51.462 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.463 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.478 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T22:57:51.487 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:51.487 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.504 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T22:57:51.512 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T22:57:51.512 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T22:57:51.527 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T22:57:51.536 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T22:57:51.536 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T22:57:51.550 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T22:57:51.559 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.559 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.576 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T22:57:51.585 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T22:57:51.585 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T22:57:51.603 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T22:57:51.612 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T22:57:51.612 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T22:57:51.629 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T22:57:51.638 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T22:57:51.638 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T22:57:51.659 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package lua5.1. 2026-02-22T22:57:51.667 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T22:57:51.667 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T22:57:51.688 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package lua-any. 2026-02-22T22:57:51.695 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T22:57:51.696 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T22:57:51.710 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package zip. 2026-02-22T22:57:51.717 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T22:57:51.718 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T22:57:51.735 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package unzip. 2026-02-22T22:57:51.742 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T22:57:51.743 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T22:57:51.759 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package luarocks. 2026-02-22T22:57:51.768 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T22:57:51.768 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T22:57:51.805 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package librgw2. 2026-02-22T22:57:51.813 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.813 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.889 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T22:57:51.898 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.898 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.914 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T22:57:51.922 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T22:57:51.922 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T22:57:51.937 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T22:57:51.946 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.946 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:51.964 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-common. 2026-02-22T22:57:51.972 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:51.973 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:52.233 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-base. 2026-02-22T22:57:52.241 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:52.246 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:52.310 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T22:57:52.318 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T22:57:52.319 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T22:57:52.334 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T22:57:52.342 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T22:57:52.342 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T22:57:52.359 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T22:57:52.368 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T22:57:52.368 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T22:57:52.382 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T22:57:52.390 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T22:57:52.391 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T22:57:52.405 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T22:57:52.414 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T22:57:52.414 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T22:57:52.428 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T22:57:52.437 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T22:57:52.437 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T22:57:52.452 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-portend. 2026-02-22T22:57:52.460 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T22:57:52.461 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T22:57:52.475 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T22:57:52.482 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T22:57:52.483 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T22:57:52.497 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T22:57:52.506 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T22:57:52.506 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T22:57:52.529 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T22:57:52.537 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T22:57:52.537 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T22:57:52.553 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T22:57:52.561 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T22:57:52.561 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T22:57:52.576 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-mako. 2026-02-22T22:57:52.584 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T22:57:52.584 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T22:57:52.603 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T22:57:52.611 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T22:57:52.612 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T22:57:52.625 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T22:57:52.633 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T22:57:52.634 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T22:57:52.647 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-webob. 2026-02-22T22:57:52.655 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T22:57:52.655 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T22:57:52.672 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T22:57:52.680 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T22:57:52.682 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T22:57:52.697 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T22:57:52.705 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T22:57:52.706 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T22:57:52.720 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-paste. 2026-02-22T22:57:52.728 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T22:57:52.729 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T22:57:52.756 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T22:57:52.764 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T22:57:52.764 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T22:57:52.779 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T22:57:52.786 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T22:57:52.787 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T22:57:52.802 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T22:57:52.810 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T22:57:52.810 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T22:57:52.826 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T22:57:52.834 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T22:57:52.834 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T22:57:52.859 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T22:57:52.867 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T22:57:52.868 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T22:57:52.887 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T22:57:52.895 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:52.895 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:52.923 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T22:57:52.931 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:52.932 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:52.947 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T22:57:52.955 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:52.955 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:52.977 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T22:57:52.985 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:52.986 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.043 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T22:57:53.051 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T22:57:53.051 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T22:57:53.067 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T22:57:53.075 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:53.076 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.231 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph. 2026-02-22T22:57:53.240 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:53.240 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.254 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T22:57:53.261 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:53.262 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.285 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T22:57:53.293 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:53.293 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.326 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package cephadm. 2026-02-22T22:57:53.335 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:53.335 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.351 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T22:57:53.359 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T22:57:53.359 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T22:57:53.381 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T22:57:53.389 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:53.390 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.409 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T22:57:53.417 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T22:57:53.417 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T22:57:53.433 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-routes. 2026-02-22T22:57:53.441 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T22:57:53.441 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T22:57:53.461 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T22:57:53.469 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:53.469 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:53.672 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T22:57:53.681 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T22:57:53.682 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T22:57:53.720 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T22:57:53.729 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T22:57:53.729 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T22:57:53.756 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T22:57:53.764 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T22:57:53.765 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T22:57:53.779 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T22:57:53.788 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T22:57:53.788 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T22:57:53.871 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T22:57:53.880 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:53.881 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:54.047 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T22:57:54.056 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T22:57:54.056 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T22:57:54.070 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T22:57:54.079 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T22:57:54.080 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T22:57:54.097 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T22:57:54.105 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T22:57:54.106 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T22:57:54.124 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T22:57:54.133 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T22:57:54.133 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T22:57:54.149 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T22:57:54.157 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T22:57:54.158 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T22:57:54.175 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T22:57:54.192 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T22:57:54.193 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T22:57:54.278 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T22:57:54.287 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:54.288 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:54.302 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T22:57:54.311 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T22:57:54.318 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T22:57:54.334 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T22:57:54.343 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T22:57:54.343 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T22:57:54.358 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package jq. 2026-02-22T22:57:54.367 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T22:57:54.367 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T22:57:54.382 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package socat. 2026-02-22T22:57:54.390 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T22:57:54.391 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T22:57:54.411 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T22:57:54.419 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T22:57:54.419 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T22:57:54.456 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-test. 2026-02-22T22:57:54.465 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:54.465 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:54.946 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T22:57:54.955 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T22:57:54.955 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:54.978 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T22:57:54.986 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:54.987 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:55.002 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T22:57:55.011 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T22:57:55.011 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T22:57:55.032 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T22:57:55.040 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T22:57:55.040 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T22:57:55.057 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T22:57:55.066 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T22:57:55.066 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T22:57:55.098 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package pkg-config. 2026-02-22T22:57:55.106 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T22:57:55.106 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T22:57:55.122 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T22:57:55.131 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T22:57:55.131 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T22:57:55.161 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T22:57:55.168 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T22:57:55.169 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T22:57:55.184 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T22:57:55.192 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T22:57:55.192 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T22:57:55.210 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T22:57:55.219 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T22:57:55.219 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T22:57:55.235 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T22:57:55.243 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T22:57:55.244 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T22:57:55.262 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-py. 2026-02-22T22:57:55.270 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T22:57:55.271 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T22:57:55.292 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T22:57:55.300 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T22:57:55.300 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T22:57:55.342 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T22:57:55.350 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T22:57:55.350 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T22:57:55.367 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-toml. 2026-02-22T22:57:55.375 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T22:57:55.375 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T22:57:55.390 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T22:57:55.399 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T22:57:55.399 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T22:57:55.421 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T22:57:55.429 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T22:57:55.429 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T22:57:55.447 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T22:57:55.455 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T22:57:55.455 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T22:57:55.522 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package radosgw. 2026-02-22T22:57:55.532 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:55.532 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:55.643 INFO:teuthology.orchestra.run.trial045.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T22:57:55.652 INFO:teuthology.orchestra.run.trial045.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T22:57:55.652 INFO:teuthology.orchestra.run.trial045.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:55.681 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T22:57:55.750 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T22:57:55.751 INFO:teuthology.orchestra.run.trial045.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T22:57:55.821 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T22:57:56.114 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T22:57:56.508 INFO:teuthology.orchestra.run.trial045.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T22:57:56.520 INFO:teuthology.orchestra.run.trial045.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:56.573 INFO:teuthology.orchestra.run.trial045.stdout:Adding system user cephadm....done 2026-02-22T22:57:56.582 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T22:57:56.654 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T22:57:56.727 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T22:57:56.728 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T22:57:56.799 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T22:57:56.874 INFO:teuthology.orchestra.run.trial045.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T22:57:56.875 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T22:57:56.969 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T22:57:57.093 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T22:57:57.169 INFO:teuthology.orchestra.run.trial045.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T22:57:57.181 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T22:57:57.259 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T22:57:57.333 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:57.400 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T22:57:57.401 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T22:57:57.402 INFO:teuthology.orchestra.run.trial045.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T22:57:57.403 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T22:57:57.404 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T22:57:57.405 INFO:teuthology.orchestra.run.trial045.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T22:57:57.421 INFO:teuthology.orchestra.run.trial045.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T22:57:57.423 INFO:teuthology.orchestra.run.trial045.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T22:57:57.424 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T22:57:57.425 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T22:57:57.546 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T22:57:57.624 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T22:57:57.707 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T22:57:57.800 INFO:teuthology.orchestra.run.trial045.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T22:57:57.801 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T22:57:58.057 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T22:57:58.135 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T22:57:58.135 INFO:teuthology.orchestra.run.trial045.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T22:57:58.136 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T22:57:58.227 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T22:57:58.352 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T22:57:58.480 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T22:57:58.563 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T22:57:58.671 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T22:57:58.743 INFO:teuthology.orchestra.run.trial045.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T22:57:58.744 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:57:58.828 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T22:57:59.334 INFO:teuthology.orchestra.run.trial045.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T22:57:59.355 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:57:59.360 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T22:57:59.440 INFO:teuthology.orchestra.run.trial045.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T22:57:59.441 INFO:teuthology.orchestra.run.trial045.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T22:57:59.442 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T22:57:59.517 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T22:57:59.590 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:57:59.591 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T22:57:59.671 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T22:57:59.746 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T22:57:59.822 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T22:57:59.897 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T22:57:59.971 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T22:58:00.049 INFO:teuthology.orchestra.run.trial045.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T22:58:00.050 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T22:58:00.135 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T22:58:00.136 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T22:58:00.214 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T22:58:00.293 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T22:58:00.379 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T22:58:00.454 INFO:teuthology.orchestra.run.trial045.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T22:58:00.455 INFO:teuthology.orchestra.run.trial045.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T22:58:00.456 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T22:58:00.456 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T22:58:00.593 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T22:58:00.671 INFO:teuthology.orchestra.run.trial045.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T22:58:00.671 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T22:58:00.743 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T22:58:00.744 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T22:58:00.828 INFO:teuthology.orchestra.run.trial045.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T22:58:00.829 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T22:58:00.910 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T22:58:01.041 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T22:58:01.131 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T22:58:01.245 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T22:58:01.246 INFO:teuthology.orchestra.run.trial045.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:01.247 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:01.248 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T22:58:01.773 INFO:teuthology.orchestra.run.trial045.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T22:58:01.784 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:01.785 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:01.786 INFO:teuthology.orchestra.run.trial045.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:01.787 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:01.788 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:01.857 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T22:58:01.857 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T22:58:02.288 INFO:teuthology.orchestra.run.trial045.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.289 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.290 INFO:teuthology.orchestra.run.trial045.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.291 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.292 INFO:teuthology.orchestra.run.trial045.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.292 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.293 INFO:teuthology.orchestra.run.trial045.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.294 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:02.322 INFO:teuthology.orchestra.run.trial045.stdout:Adding group ceph....done 2026-02-22T22:58:02.358 INFO:teuthology.orchestra.run.trial045.stdout:Adding system user ceph....done 2026-02-22T22:58:02.368 INFO:teuthology.orchestra.run.trial045.stdout:Setting system user ceph properties....done 2026-02-22T22:58:02.374 INFO:teuthology.orchestra.run.trial045.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T22:58:02.443 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T22:58:02.717 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T22:58:03.102 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:03.102 INFO:teuthology.orchestra.run.trial045.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:03.367 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T22:58:03.368 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T22:58:03.747 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:03.861 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T22:58:04.248 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:04.308 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T22:58:04.308 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T22:58:04.697 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:04.756 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T22:58:04.756 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T22:58:05.172 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:05.254 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T22:58:05.255 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T22:58:05.656 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:05.657 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:05.667 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:05.738 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T22:58:05.738 INFO:teuthology.orchestra.run.trial045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T22:58:06.156 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:06.168 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:06.169 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:06.181 INFO:teuthology.orchestra.run.trial045.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T22:58:06.291 INFO:teuthology.orchestra.run.trial045.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T22:58:06.298 INFO:teuthology.orchestra.run.trial045.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T22:58:06.541 INFO:teuthology.orchestra.run.trial045.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T22:58:06.625 INFO:teuthology.orchestra.run.trial045.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T22:58:07.136 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:07.137 INFO:teuthology.orchestra.run.trial045.stdout:Running kernel seems to be up-to-date. 2026-02-22T22:58:07.137 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:07.137 INFO:teuthology.orchestra.run.trial045.stdout:The processor microcode seems to be up-to-date. 2026-02-22T22:58:07.137 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:07.137 INFO:teuthology.orchestra.run.trial045.stdout:No services need to be restarted. 2026-02-22T22:58:07.137 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:07.137 INFO:teuthology.orchestra.run.trial045.stdout:No containers need to be restarted. 2026-02-22T22:58:07.138 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:07.138 INFO:teuthology.orchestra.run.trial045.stdout:No user sessions are running outdated binaries. 2026-02-22T22:58:07.138 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:07.138 INFO:teuthology.orchestra.run.trial045.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T22:58:07.953 INFO:teuthology.orchestra.run.trial045.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T22:58:07.969 DEBUG:teuthology.parallel:result is None 2026-02-22T22:58:07.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:58:08.149 DEBUG:teuthology.orchestra.run.trial045:> dpkg-query -W -f '${Version}' ceph 2026-02-22T22:58:08.160 INFO:teuthology.orchestra.run.trial045.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:58:08.161 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T22:58:08.161 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T22:58:08.164 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T22:58:08.164 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:08.164 DEBUG:teuthology.orchestra.run.trial045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T22:58:08.209 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T22:58:08.210 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:08.210 DEBUG:teuthology.orchestra.run.trial045:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T22:58:08.263 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T22:58:08.313 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T22:58:08.313 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:08.313 DEBUG:teuthology.orchestra.run.trial045:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T22:58:08.362 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T22:58:08.413 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T22:58:08.413 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:08.413 DEBUG:teuthology.orchestra.run.trial045:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T22:58:08.466 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T22:58:08.518 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T22:58:08.640 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-22T22:58:08.640 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod 777 /var/log/ceph 2026-02-22T22:58:08.648 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-22T22:58:08.648 DEBUG:teuthology.orchestra.run.trial045:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T22:58:08.696 INFO:tasks.ceph:Creating extra log directories... 2026-02-22T22:58:08.697 DEBUG:teuthology.orchestra.run.trial045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T22:58:08.756 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-22T22:58:08.756 INFO:tasks.ceph:config {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T22:58:08.757 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '64311', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5907, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+pqmRDeK/i+ZrJqFYf4dHmFkLOFO7XJRfTTRa4EIPYeI6K11h1L+r4wDMbslfvUPqBS1cmhnTN9r6IvawHoQQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['cephtool/test_kvstore_tool.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:40', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T22:58:08.757 DEBUG:teuthology.orchestra.run.trial045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-22T22:58:08.793 DEBUG:teuthology.orchestra.run.trial045:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T22:58:08.845 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:08.845 DEBUG:teuthology.orchestra.run.trial045:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T22:58:08.893 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T22:58:08.893 DEBUG:teuthology.orchestra.run.trial045:> stat /dev/vg_nvme/lv_1 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout:Device: 5h/5d Inode: 905 Links: 1 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout:Access: 2026-02-22 22:57:24.823621620 +0000 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout:Modify: 2026-02-22 22:57:24.651624544 +0000 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout:Change: 2026-02-22 22:57:24.651624544 +0000 2026-02-22T22:58:08.940 INFO:teuthology.orchestra.run.trial045.stdout: Birth: - 2026-02-22T22:58:08.940 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T22:58:08.987 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records in 2026-02-22T22:58:08.987 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records out 2026-02-22T22:58:08.987 INFO:teuthology.orchestra.run.trial045.stderr:512 bytes copied, 0.000195102 s, 2.6 MB/s 2026-02-22T22:58:08.988 DEBUG:teuthology.orchestra.run.trial045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T22:58:09.033 DEBUG:teuthology.orchestra.run.trial045:> stat /dev/vg_nvme/lv_2 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout:Device: 5h/5d Inode: 943 Links: 1 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout:Access: 2026-02-22 22:57:24.999618628 +0000 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout:Modify: 2026-02-22 22:57:24.995618693 +0000 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout:Change: 2026-02-22 22:57:24.995618693 +0000 2026-02-22T22:58:09.080 INFO:teuthology.orchestra.run.trial045.stdout: Birth: - 2026-02-22T22:58:09.080 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T22:58:09.127 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records in 2026-02-22T22:58:09.127 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records out 2026-02-22T22:58:09.127 INFO:teuthology.orchestra.run.trial045.stderr:512 bytes copied, 0.000173903 s, 2.9 MB/s 2026-02-22T22:58:09.128 DEBUG:teuthology.orchestra.run.trial045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T22:58:09.173 DEBUG:teuthology.orchestra.run.trial045:> stat /dev/vg_nvme/lv_3 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout:Device: 5h/5d Inode: 966 Links: 1 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout:Access: 2026-02-22 22:57:25.371612301 +0000 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout:Modify: 2026-02-22 22:57:25.367612371 +0000 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout:Change: 2026-02-22 22:57:25.367612371 +0000 2026-02-22T22:58:09.224 INFO:teuthology.orchestra.run.trial045.stdout: Birth: - 2026-02-22T22:58:09.224 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T22:58:09.272 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records in 2026-02-22T22:58:09.272 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records out 2026-02-22T22:58:09.273 INFO:teuthology.orchestra.run.trial045.stderr:512 bytes copied, 0.000179613 s, 2.9 MB/s 2026-02-22T22:58:09.273 DEBUG:teuthology.orchestra.run.trial045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T22:58:09.327 DEBUG:teuthology.orchestra.run.trial045:> stat /dev/vg_nvme/lv_4 2026-02-22T22:58:09.377 INFO:teuthology.orchestra.run.trial045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T22:58:09.377 INFO:teuthology.orchestra.run.trial045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T22:58:09.377 INFO:teuthology.orchestra.run.trial045.stdout:Device: 5h/5d Inode: 978 Links: 1 2026-02-22T22:58:09.377 INFO:teuthology.orchestra.run.trial045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T22:58:09.377 INFO:teuthology.orchestra.run.trial045.stdout:Access: 2026-02-22 22:57:25.891603463 +0000 2026-02-22T22:58:09.378 INFO:teuthology.orchestra.run.trial045.stdout:Modify: 2026-02-22 22:57:25.727606251 +0000 2026-02-22T22:58:09.378 INFO:teuthology.orchestra.run.trial045.stdout:Change: 2026-02-22 22:57:25.727606251 +0000 2026-02-22T22:58:09.378 INFO:teuthology.orchestra.run.trial045.stdout: Birth: - 2026-02-22T22:58:09.378 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T22:58:09.428 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records in 2026-02-22T22:58:09.428 INFO:teuthology.orchestra.run.trial045.stderr:1+0 records out 2026-02-22T22:58:09.428 INFO:teuthology.orchestra.run.trial045.stderr:512 bytes copied, 0.00016137 s, 3.2 MB/s 2026-02-22T22:58:09.429 DEBUG:teuthology.orchestra.run.trial045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T22:58:09.478 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:58:09.478 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T22:58:09.478 INFO:tasks.ceph:Generating config... 2026-02-22T22:58:09.481 INFO:tasks.ceph:[global] mon election default strategy = 3 2026-02-22T22:58:09.481 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T22:58:09.481 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T22:58:09.481 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T22:58:09.482 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T22:58:09.482 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T22:58:09.482 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-22T22:58:09.482 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-22T22:58:09.483 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-22T22:58:09.483 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-22T22:58:09.483 INFO:tasks.ceph:Setting up mon.a... 2026-02-22T22:58:09.483 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-22T22:58:09.537 INFO:teuthology.orchestra.run.trial045.stdout:creating /etc/ceph/ceph.keyring 2026-02-22T22:58:09.540 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-22T22:58:09.604 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T22:58:09.656 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.45')] 2026-02-22T22:58:09.657 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.45', 'mon election default strategy': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-22T22:58:09.658 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:09.659 DEBUG:teuthology.orchestra.run.trial045:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T22:58:09.701 DEBUG:teuthology.orchestra.run.trial045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.45 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:58:09.762 INFO:teuthology.orchestra.run.trial045.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:58:09.762 INFO:teuthology.orchestra.run.trial045.stdout:monmaptool: generated fsid 63f08d12-d287-4276-9f47-af125eb32eff 2026-02-22T22:58:09.762 INFO:teuthology.orchestra.run.trial045.stdout:setting min_mon_release = quincy 2026-02-22T22:58:09.762 INFO:teuthology.orchestra.run.trial045.stdout:epoch 0 2026-02-22T22:58:09.762 INFO:teuthology.orchestra.run.trial045.stdout:fsid 63f08d12-d287-4276-9f47-af125eb32eff 2026-02-22T22:58:09.762 INFO:teuthology.orchestra.run.trial045.stdout:last_changed 2026-02-22T22:58:09.759040+0000 2026-02-22T22:58:09.763 INFO:teuthology.orchestra.run.trial045.stdout:created 2026-02-22T22:58:09.759040+0000 2026-02-22T22:58:09.763 INFO:teuthology.orchestra.run.trial045.stdout:min_mon_release 17 (quincy) 2026-02-22T22:58:09.763 INFO:teuthology.orchestra.run.trial045.stdout:election_strategy: 3 2026-02-22T22:58:09.763 INFO:teuthology.orchestra.run.trial045.stdout:0: [v2:10.20.193.45:3300/0,v1:10.20.193.45:6789/0] mon.a 2026-02-22T22:58:09.763 INFO:teuthology.orchestra.run.trial045.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-22T22:58:09.764 DEBUG:teuthology.orchestra.run.trial045:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T22:58:09.809 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 63f08d12-d287-4276-9f47-af125eb32eff... 2026-02-22T22:58:09.811 DEBUG:teuthology.orchestra.run.trial045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-22T22:58:09.868 INFO:teuthology.orchestra.run.trial045.stdout:[global] 2026-02-22T22:58:09.868 INFO:teuthology.orchestra.run.trial045.stdout: chdir = "" 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: auth supported = cephx 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: filestore xattr use omap = true 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: mon clock drift allowed = 1.000 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: osd crush chooseleaf type = 0 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: auth debug = true 2026-02-22T22:58:09.869 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: ms die on old message = true 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: ms die on bug = true 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: mon pg warn max object skew = 0 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: osd pool default size = 2 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.870 INFO:teuthology.orchestra.run.trial045.stdout: mon osd allow primary affinity = true 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon osd allow pg remap = true 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon warn on legacy crush tunables = false 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon warn on crush straw calc version zero = false 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon warn on no sortbitwise = false 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon warn on osd down out interval zero = false 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon warn on too few osds = false 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: mon_allow_pool_size_one = true 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T22:58:09.871 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: osd default data pool replay window = 5 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: mon allow pool delete = true 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: mon cluster log file level = debug 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: debug asserts on shutdown = true 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: mon health detail to clog = false 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: mon host = 10.20.193.45 2026-02-22T22:58:09.872 INFO:teuthology.orchestra.run.trial045.stdout: mon election default strategy = 3 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: fsid = 63f08d12-d287-4276-9f47-af125eb32eff 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout:[osd] 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: osd journal size = 100 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: osd scrub load threshold = 5.0 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: osd scrub max interval = 600 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: osd mclock profile = high_recovery_ops 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: osd mclock skip benchmark = True 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: osd recover clone overlap = true 2026-02-22T22:58:09.873 INFO:teuthology.orchestra.run.trial045.stdout: osd recovery max chunk = 1048576 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd debug shutdown = true 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd debug op order = true 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd debug verify stray on activate = true 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd debug trim objects = true 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd open classes on start = true 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd debug pg log writeout = true 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd deep scrub update digest min age = 30 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: osd map max advance = 10 2026-02-22T22:58:09.874 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: journal zero on create = true 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: filestore ondisk finisher threads = 3 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: filestore apply finisher threads = 3 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: bdev debug aio = true 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: osd debug misdirected ops = true 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: bluestore zero block detection = True 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: debug ms = 1 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: debug osd = 20 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: osd debug verify cached snaps = True 2026-02-22T22:58:09.875 INFO:teuthology.orchestra.run.trial045.stdout: osd debug verify missing on start = True 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: osd mclock override recovery settings = True 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: osd op queue = debug_random 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: osd op queue cut off = debug_random 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout:[mgr] 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: debug ms = 1 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: debug mgr = 20 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: debug mon = 20 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: debug auth = 20 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: mon reweight min pgs per osd = 4 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: mon reweight min bytes per osd = 10 2026-02-22T22:58:09.876 INFO:teuthology.orchestra.run.trial045.stdout: mgr/telemetry/nag = false 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout:[mon] 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: debug ms = 1 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: debug mon = 20 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: debug paxos = 20 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: debug auth = 20 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: mon data avail warn = 5 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: mon mgr mkfs grace = 240 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: mon reweight min pgs per osd = 4 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: mon osd reporter subtree level = osd 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: mon osd prime pg temp = true 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: mon reweight min bytes per osd = 10 2026-02-22T22:58:09.877 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: # 1m isn't quite enough 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: mon_down_mkfs_grace = 2m 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.878 INFO:teuthology.orchestra.run.trial045.stdout: mon_warn_on_filestore_osds = false 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout: mon scrub interval = 300 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout:[client] 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout: rgw cache enabled = true 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout: rgw enable ops log = true 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout: rgw enable usage log = true 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T22:58:09.879 INFO:teuthology.orchestra.run.trial045.stdout:[mon.a] 2026-02-22T22:58:09.880 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-22T22:58:09.881 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-22T22:58:09.944 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T22:58:09.944 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:09.944 DEBUG:teuthology.orchestra.run.trial045:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-22T22:58:09.992 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:09.993 DEBUG:teuthology.orchestra.run.trial045:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-22T22:58:10.036 INFO:tasks.ceph:Sending monmap to node ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:58:10.036 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:10.036 DEBUG:teuthology.orchestra.run.trial045:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-22T22:58:10.037 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T22:58:10.090 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:10.091 DEBUG:teuthology.orchestra.run.trial045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:58:10.137 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T22:58:10.137 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T22:58:10.137 DEBUG:teuthology.orchestra.run.trial045:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T22:58:10.209 INFO:teuthology.orchestra.run.trial045.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T22:58:10.212 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T22:58:10.212 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T22:58:10.213 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-22T22:58:10.275 INFO:teuthology.orchestra.run.trial045.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-22T22:58:10.285 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T22:58:10.285 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial045.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-22T22:58:10.285 DEBUG:teuthology.orchestra.run.trial045:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-22T22:58:10.337 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:58:10.337 INFO:tasks.ceph:role: osd.0 2026-02-22T22:58:10.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:58:10.337 DEBUG:teuthology.orchestra.run.trial045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T22:58:10.386 INFO:teuthology.orchestra.run.trial045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T22:58:10.386 INFO:teuthology.orchestra.run.trial045.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T22:58:10.386 INFO:teuthology.orchestra.run.trial045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T22:58:10.386 INFO:teuthology.orchestra.run.trial045.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T22:58:10.386 INFO:teuthology.orchestra.run.trial045.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T22:58:10.386 INFO:teuthology.orchestra.run.trial045.stdout: = sunit=0 swidth=0 blks 2026-02-22T22:58:10.386 INFO:teuthology.orchestra.run.trial045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T22:58:10.387 INFO:teuthology.orchestra.run.trial045.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T22:58:10.387 INFO:teuthology.orchestra.run.trial045.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T22:58:10.387 INFO:teuthology.orchestra.run.trial045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T22:58:10.980 INFO:teuthology.orchestra.run.trial045.stdout:Discarding blocks...Done. 2026-02-22T22:58:10.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial045.front.sepia.ceph.com -o noatime 2026-02-22T22:58:10.982 DEBUG:teuthology.orchestra.run.trial045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-22T22:58:11.036 DEBUG:teuthology.orchestra.run.trial045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-22T22:58:11.083 INFO:teuthology.orchestra.run.trial045.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T22:58:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:58:11.084 DEBUG:teuthology.orchestra.run.trial045:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-22T22:58:11.132 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:58:11.132 INFO:tasks.ceph:role: osd.1 2026-02-22T22:58:11.132 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:58:11.132 DEBUG:teuthology.orchestra.run.trial045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout: = sunit=0 swidth=0 blks 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T22:58:11.182 INFO:teuthology.orchestra.run.trial045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T22:58:11.752 INFO:teuthology.orchestra.run.trial045.stdout:Discarding blocks...Done. 2026-02-22T22:58:11.754 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial045.front.sepia.ceph.com -o noatime 2026-02-22T22:58:11.754 DEBUG:teuthology.orchestra.run.trial045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-22T22:58:11.810 DEBUG:teuthology.orchestra.run.trial045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-22T22:58:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:58:11.859 INFO:teuthology.orchestra.run.trial045.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T22:58:11.860 DEBUG:teuthology.orchestra.run.trial045:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-22T22:58:11.909 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-22T22:58:11.909 INFO:tasks.ceph:role: osd.2 2026-02-22T22:58:11.909 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:58:11.910 DEBUG:teuthology.orchestra.run.trial045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T22:58:11.962 INFO:teuthology.orchestra.run.trial045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T22:58:11.962 INFO:teuthology.orchestra.run.trial045.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T22:58:11.962 INFO:teuthology.orchestra.run.trial045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T22:58:11.962 INFO:teuthology.orchestra.run.trial045.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T22:58:11.963 INFO:teuthology.orchestra.run.trial045.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T22:58:11.963 INFO:teuthology.orchestra.run.trial045.stdout: = sunit=0 swidth=0 blks 2026-02-22T22:58:11.963 INFO:teuthology.orchestra.run.trial045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T22:58:11.963 INFO:teuthology.orchestra.run.trial045.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T22:58:11.963 INFO:teuthology.orchestra.run.trial045.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T22:58:11.963 INFO:teuthology.orchestra.run.trial045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T22:58:12.542 INFO:teuthology.orchestra.run.trial045.stdout:Discarding blocks...Done. 2026-02-22T22:58:12.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial045.front.sepia.ceph.com -o noatime 2026-02-22T22:58:12.545 DEBUG:teuthology.orchestra.run.trial045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-22T22:58:12.602 DEBUG:teuthology.orchestra.run.trial045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-22T22:58:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:58:12.651 INFO:teuthology.orchestra.run.trial045.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T22:58:12.652 DEBUG:teuthology.orchestra.run.trial045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:58:12.713 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:12.709+0000 7fd3016338c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-22T22:58:12.714 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:12.709+0000 7fd3016338c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-22T22:58:12.714 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:12.709+0000 7fd3016338c0 -1 bdev(0x55c17c115800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-22T22:58:12.714 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:12.709+0000 7fd3016338c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-22T22:58:15.029 DEBUG:teuthology.orchestra.run.trial045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-22T22:58:15.077 DEBUG:teuthology.orchestra.run.trial045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:58:15.141 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:15.137+0000 7fb9c3f368c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-22T22:58:15.142 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:15.137+0000 7fb9c3f368c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-22T22:58:15.143 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:15.137+0000 7fb9c3f368c0 -1 bdev(0x56068e371800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-22T22:58:15.143 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:15.137+0000 7fb9c3f368c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-22T22:58:17.436 DEBUG:teuthology.orchestra.run.trial045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-22T22:58:17.490 DEBUG:teuthology.orchestra.run.trial045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:58:17.553 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:17.549+0000 7fb0394358c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-22T22:58:17.554 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:17.549+0000 7fb0394358c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-22T22:58:17.554 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:17.549+0000 7fb0394358c0 -1 bdev(0x55a67a44b800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-22T22:58:17.554 INFO:teuthology.orchestra.run.trial045.stderr:2026-02-22T22:58:17.549+0000 7fb0394358c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-22T22:58:19.832 DEBUG:teuthology.orchestra.run.trial045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-22T22:58:19.882 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T22:58:19.882 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:19.882 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-22T22:58:19.934 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:19.934 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-22T22:58:19.986 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:19.986 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-22T22:58:20.038 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:20.038 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-22T22:58:20.090 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:20.091 DEBUG:teuthology.orchestra.run.trial045:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-22T22:58:20.137 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T22:58:20.138 DEBUG:teuthology.orchestra.run.trial045:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-22T22:58:20.187 INFO:teuthology.orchestra.run.trial045.stdout:[mgr.x] 2026-02-22T22:58:20.187 INFO:teuthology.orchestra.run.trial045.stdout: key = AQACiptpLFxKDBAADFa1BRa4hmPT92viAk6FBg== 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout:[osd.0] 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout: key = AQAEiptpCutmKhAAOIZdgOAZ4uealzW9QD6HcA== 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout:[osd.1] 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout: key = AQAHiptpG+xLCBAATlUf9Mmpgqn28SErdKSkdg== 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout:[osd.2] 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout: key = AQAJiptpmUHTIBAA5XS9FbYLL3yJXZgPVKlkdg== 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout:[client.0] 2026-02-22T22:58:20.188 INFO:teuthology.orchestra.run.trial045.stdout: key = AQACiptp2EJJEBAAngUBwq+c0D/FPA0C4U7N7Q== 2026-02-22T22:58:20.190 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T22:58:20.256 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T22:58:20.318 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T22:58:20.383 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T22:58:20.446 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T22:58:20.511 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T22:58:20.511 DEBUG:teuthology.orchestra.run.trial045:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-22T22:58:20.560 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T22:58:20.626 DEBUG:teuthology.orchestra.run.trial045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-22T22:58:20.677 DEBUG:teuthology.orchestra.run.trial045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-22T22:58:20.725 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-22T22:58:20.725 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-22T22:58:20.726 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-22T22:58:20.770 INFO:tasks.ceph.mon.a:Started 2026-02-22T22:58:20.771 DEBUG:teuthology.orchestra.run.trial045:> sudo ceph config set mgr mgr_pool false --force 2026-02-22T22:58:20.899 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-22T22:58:20.900 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-22T22:58:20.900 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-22T22:58:20.902 INFO:tasks.ceph.mgr.x:Started 2026-02-22T22:58:20.902 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T22:58:20.902 DEBUG:teuthology.orchestra.run.trial045:> sudo ceph --cluster ceph osd crush tunables default 2026-02-22T22:58:21.008 INFO:teuthology.orchestra.run.trial045.stderr:adjusted tunables profile to default 2026-02-22T22:58:21.009 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.005+0000 7fd988db4280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T22:58:21.018 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T22:58:21.018 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-22T22:58:21.018 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:21.018 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-22T22:58:21.026 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:21.027 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-22T22:58:21.035 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.029+0000 7fd988db4280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T22:58:21.042 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:58:21.042 DEBUG:teuthology.orchestra.run.trial045:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-22T22:58:21.074 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.069+0000 7fd988db4280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T22:58:21.081 DEBUG:teuthology.orchestra.run.trial045:> sudo ceph --cluster ceph osd new 1f888aa2-0eec-4e5e-8241-a48a22dd208d 0 2026-02-22T22:58:21.144 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.137+0000 7fd988db4280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T22:58:21.222 INFO:teuthology.orchestra.run.trial045.stdout:0 2026-02-22T22:58:21.224 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.217+0000 7fd988db4280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T22:58:21.231 DEBUG:teuthology.orchestra.run.trial045:> sudo ceph --cluster ceph osd new 6981b8b6-8c9a-44d8-a43f-0a68ccdb5037 1 2026-02-22T22:58:21.262 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.257+0000 7fd988db4280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T22:58:21.337 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.329+0000 7fd988db4280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T22:58:21.357 INFO:teuthology.orchestra.run.trial045.stdout:1 2026-02-22T22:58:21.366 DEBUG:teuthology.orchestra.run.trial045:> sudo ceph --cluster ceph osd new 82cd9280-8291-4ad9-a089-6fc67799bcbf 2 2026-02-22T22:58:21.466 INFO:teuthology.orchestra.run.trial045.stdout:2 2026-02-22T22:58:21.476 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-22T22:58:21.476 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-22T22:58:21.478 INFO:tasks.ceph.osd.0:Started 2026-02-22T22:58:21.479 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-22T22:58:21.479 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-22T22:58:21.481 INFO:tasks.ceph.osd.1:Started 2026-02-22T22:58:21.482 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-22T22:58:21.482 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-22T22:58:21.484 INFO:tasks.ceph.osd.2:Started 2026-02-22T22:58:21.484 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:58:21.515 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.509+0000 7fd988db4280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T22:58:21.578 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:21.578 INFO:teuthology.orchestra.run.trial045.stdout:{"epoch":5,"fsid":"63f08d12-d287-4276-9f47-af125eb32eff","created":"2026-02-22T22:58:20.847724+0000","modified":"2026-02-22T22:58:21.463222+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T22:58:21.463222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1f888aa2-0eec-4e5e-8241-a48a22dd208d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6981b8b6-8c9a-44d8-a43f-0a68ccdb5037","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"82cd9280-8291-4ad9-a089-6fc67799bcbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:58:21.587 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-22T22:58:21.587 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T22:58:21.819 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.813+0000 7fd988db4280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T22:58:21.888 DEBUG:teuthology.orchestra.run.trial045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T22:58:21.912 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.905+0000 7fd988db4280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T22:58:21.940 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.933+0000 7fd988db4280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T22:58:21.972 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:21.965+0000 7fd988db4280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T22:58:21.976 INFO:teuthology.misc.health.trial045.stdout: 2026-02-22T22:58:21.976 INFO:teuthology.misc.health.trial045.stdout:{"epoch":5,"fsid":"63f08d12-d287-4276-9f47-af125eb32eff","created":"2026-02-22T22:58:20.847724+0000","modified":"2026-02-22T22:58:21.463222+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T22:58:21.463222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1f888aa2-0eec-4e5e-8241-a48a22dd208d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6981b8b6-8c9a-44d8-a43f-0a68ccdb5037","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"82cd9280-8291-4ad9-a089-6fc67799bcbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:58:21.984 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T22:58:22.016 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.009+0000 7fd988db4280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T22:58:22.121 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.113+0000 7fd988db4280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T22:58:22.317 INFO:tasks.ceph.mgr.x.trial045.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T22:58:22.494 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.489+0000 7fd988db4280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T22:58:22.527 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.521+0000 7fd988db4280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T22:58:22.622 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.617+0000 7fd988db4280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T22:58:22.646 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.641+0000 7fd988db4280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T22:58:22.673 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.669+0000 7fd988db4280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T22:58:22.701 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.697+0000 7fd988db4280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T22:58:22.822 INFO:tasks.ceph.mgr.x.trial045.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T22:58:22.822 INFO:tasks.ceph.mgr.x.trial045.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T22:58:22.822 INFO:tasks.ceph.mgr.x.trial045.stderr: from numpy import show_config as show_numpy_config 2026-02-22T22:58:22.829 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.825+0000 7fd988db4280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T22:58:22.856 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.849+0000 7fd988db4280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T22:58:22.896 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:22.889+0000 7fd988db4280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T22:58:23.086 INFO:tasks.ceph.osd.2.trial045.stderr:2026-02-22T22:58:23.081+0000 7fbea2eea8c0 -1 Falling back to public interface 2026-02-22T22:58:23.510 INFO:tasks.ceph.osd.0.trial045.stderr:2026-02-22T22:58:23.505+0000 7fcaf17b08c0 -1 Falling back to public interface 2026-02-22T22:58:23.534 INFO:tasks.ceph.osd.1.trial045.stderr:2026-02-22T22:58:23.529+0000 7ff5518f18c0 -1 Falling back to public interface 2026-02-22T22:58:23.924 INFO:tasks.ceph.mgr.x.trial045.stderr:2026-02-22T22:58:23.917+0000 7fd8a7601640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-22T22:58:26.079 INFO:tasks.ceph.osd.2.trial045.stderr:2026-02-22T22:58:26.073+0000 7fbea2eea8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:58:26.711 INFO:tasks.ceph.osd.0.trial045.stderr:2026-02-22T22:58:26.705+0000 7fcaf17b08c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:58:26.809 INFO:tasks.ceph.osd.1.trial045.stderr:2026-02-22T22:58:26.805+0000 7ff5518f18c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:58:27.927 INFO:tasks.ceph.osd.1.trial045.stderr:2026-02-22T22:58:27.921+0000 7ff54d89a640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T22:58:27.927 INFO:tasks.ceph.osd.0.trial045.stderr:2026-02-22T22:58:27.921+0000 7fcaed759640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T22:58:27.928 INFO:tasks.ceph.osd.2.trial045.stderr:2026-02-22T22:58:27.921+0000 7fbe9ee93640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T22:58:27.950 INFO:tasks.ceph.osd.0.trial045.stderr:2026-02-22T22:58:27.945+0000 7fcae9594640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:58:27.951 INFO:tasks.ceph.osd.1.trial045.stderr:2026-02-22T22:58:27.945+0000 7ff5496d5640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:58:27.951 INFO:tasks.ceph.osd.2.trial045.stderr:2026-02-22T22:58:27.945+0000 7fbe9acce640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T22:58:28.287 DEBUG:teuthology.orchestra.run.trial045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T22:58:28.428 INFO:teuthology.misc.health.trial045.stdout: 2026-02-22T22:58:28.428 INFO:teuthology.misc.health.trial045.stdout:{"epoch":7,"fsid":"63f08d12-d287-4276-9f47-af125eb32eff","created":"2026-02-22T22:58:20.847724+0000","modified":"2026-02-22T22:58:27.919988+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T22:58:21.463222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1f888aa2-0eec-4e5e-8241-a48a22dd208d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6981b8b6-8c9a-44d8-a43f-0a68ccdb5037","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"82cd9280-8291-4ad9-a089-6fc67799bcbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:58:28.437 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-22T22:58:34.740 DEBUG:teuthology.orchestra.run.trial045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T22:58:34.882 INFO:teuthology.misc.health.trial045.stdout: 2026-02-22T22:58:34.882 INFO:teuthology.misc.health.trial045.stdout:{"epoch":8,"fsid":"63f08d12-d287-4276-9f47-af125eb32eff","created":"2026-02-22T22:58:20.847724+0000","modified":"2026-02-22T22:58:28.923362+0000","last_up_change":"2026-02-22T22:58:28.923362+0000","last_in_change":"2026-02-22T22:58:21.463222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1f888aa2-0eec-4e5e-8241-a48a22dd208d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6810","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6811","nonce":4207577072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6812","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6813","nonce":4207577072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6816","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6817","nonce":4207577072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6814","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6815","nonce":4207577072}]},"public_addr":"10.20.193.45:6811/4207577072","cluster_addr":"10.20.193.45:6813/4207577072","heartbeat_back_addr":"10.20.193.45:6817/4207577072","heartbeat_front_addr":"10.20.193.45:6815/4207577072","state":["exists","up"]},{"osd":1,"uuid":"6981b8b6-8c9a-44d8-a43f-0a68ccdb5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6818","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6819","nonce":2231773529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6820","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6821","nonce":2231773529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6824","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6825","nonce":2231773529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6822","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6823","nonce":2231773529}]},"public_addr":"10.20.193.45:6819/2231773529","cluster_addr":"10.20.193.45:6821/2231773529","heartbeat_back_addr":"10.20.193.45:6825/2231773529","heartbeat_front_addr":"10.20.193.45:6823/2231773529","state":["exists","up"]},{"osd":2,"uuid":"82cd9280-8291-4ad9-a089-6fc67799bcbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6802","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6803","nonce":25824641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6804","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6805","nonce":25824641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6808","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6809","nonce":25824641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6806","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6807","nonce":25824641}]},"public_addr":"10.20.193.45:6803/25824641","cluster_addr":"10.20.193.45:6805/25824641","heartbeat_back_addr":"10.20.193.45:6809/25824641","heartbeat_front_addr":"10.20.193.45:6807/25824641","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:58:34.891 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-22T22:58:34.891 INFO:tasks.ceph:Creating RBD pool 2026-02-22T22:58:34.891 DEBUG:teuthology.orchestra.run.trial045:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-22T22:58:35.936 INFO:teuthology.orchestra.run.trial045.stderr:pool 'rbd' created 2026-02-22T22:58:35.950 DEBUG:teuthology.orchestra.run.trial045:> rbd --cluster ceph pool init rbd 2026-02-22T22:58:38.951 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-22T22:58:38.952 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-22T22:58:38.952 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-22T22:58:38.952 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-22T22:58:38.953 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T22:58:39.124 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:39.134 INFO:teuthology.orchestra.run.trial045.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6800","nonce":3932992620},{"type":"v1","addr":"10.20.193.45:6801","nonce":3932992620}]},"active_addr":"10.20.193.45:6801/3932992620","active_change":"2026-02-22T22:58:22.915351+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.45:0","nonce":481379580}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.45:0","nonce":2066235780}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.45:0","nonce":3944996150}]}]} 2026-02-22T22:58:39.137 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-22T22:58:39.138 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-22T22:58:39.138 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:58:39.289 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:39.290 INFO:teuthology.orchestra.run.trial045.stdout:{"epoch":12,"fsid":"63f08d12-d287-4276-9f47-af125eb32eff","created":"2026-02-22T22:58:20.847724+0000","modified":"2026-02-22T22:58:38.938154+0000","last_up_change":"2026-02-22T22:58:28.923362+0000","last_in_change":"2026-02-22T22:58:21.463222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-22T22:58:35.038338+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1f888aa2-0eec-4e5e-8241-a48a22dd208d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6810","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6811","nonce":4207577072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6812","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6813","nonce":4207577072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6816","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6817","nonce":4207577072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6814","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6815","nonce":4207577072}]},"public_addr":"10.20.193.45:6811/4207577072","cluster_addr":"10.20.193.45:6813/4207577072","heartbeat_back_addr":"10.20.193.45:6817/4207577072","heartbeat_front_addr":"10.20.193.45:6815/4207577072","state":["exists","up"]},{"osd":1,"uuid":"6981b8b6-8c9a-44d8-a43f-0a68ccdb5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6818","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6819","nonce":2231773529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6820","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6821","nonce":2231773529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6824","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6825","nonce":2231773529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6822","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6823","nonce":2231773529}]},"public_addr":"10.20.193.45:6819/2231773529","cluster_addr":"10.20.193.45:6821/2231773529","heartbeat_back_addr":"10.20.193.45:6825/2231773529","heartbeat_front_addr":"10.20.193.45:6823/2231773529","state":["exists","up"]},{"osd":2,"uuid":"82cd9280-8291-4ad9-a089-6fc67799bcbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6802","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6803","nonce":25824641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6804","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6805","nonce":25824641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6808","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6809","nonce":25824641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6806","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6807","nonce":25824641}]},"public_addr":"10.20.193.45:6803/25824641","cluster_addr":"10.20.193.45:6805/25824641","heartbeat_back_addr":"10.20.193.45:6809/25824641","heartbeat_front_addr":"10.20.193.45:6807/25824641","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:58:39.300 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-22T22:58:39.300 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:58:39.443 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:39.443 INFO:teuthology.orchestra.run.trial045.stdout:{"epoch":12,"fsid":"63f08d12-d287-4276-9f47-af125eb32eff","created":"2026-02-22T22:58:20.847724+0000","modified":"2026-02-22T22:58:38.938154+0000","last_up_change":"2026-02-22T22:58:28.923362+0000","last_in_change":"2026-02-22T22:58:21.463222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-22T22:58:35.038338+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1f888aa2-0eec-4e5e-8241-a48a22dd208d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6810","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6811","nonce":4207577072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6812","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6813","nonce":4207577072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6816","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6817","nonce":4207577072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6814","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6815","nonce":4207577072}]},"public_addr":"10.20.193.45:6811/4207577072","cluster_addr":"10.20.193.45:6813/4207577072","heartbeat_back_addr":"10.20.193.45:6817/4207577072","heartbeat_front_addr":"10.20.193.45:6815/4207577072","state":["exists","up"]},{"osd":1,"uuid":"6981b8b6-8c9a-44d8-a43f-0a68ccdb5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6818","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6819","nonce":2231773529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6820","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6821","nonce":2231773529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6824","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6825","nonce":2231773529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6822","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6823","nonce":2231773529}]},"public_addr":"10.20.193.45:6819/2231773529","cluster_addr":"10.20.193.45:6821/2231773529","heartbeat_back_addr":"10.20.193.45:6825/2231773529","heartbeat_front_addr":"10.20.193.45:6823/2231773529","state":["exists","up"]},{"osd":2,"uuid":"82cd9280-8291-4ad9-a089-6fc67799bcbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6802","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6803","nonce":25824641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6804","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6805","nonce":25824641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6808","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6809","nonce":25824641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6806","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6807","nonce":25824641}]},"public_addr":"10.20.193.45:6803/25824641","cluster_addr":"10.20.193.45:6805/25824641","heartbeat_back_addr":"10.20.193.45:6809/25824641","heartbeat_front_addr":"10.20.193.45:6807/25824641","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:58:39.454 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T22:58:39.455 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T22:58:39.455 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T22:58:39.538 INFO:teuthology.orchestra.run.trial045.stdout:34359738372 2026-02-22T22:58:39.539 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T22:58:39.540 INFO:teuthology.orchestra.run.trial045.stdout:34359738371 2026-02-22T22:58:39.541 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T22:58:39.542 INFO:teuthology.orchestra.run.trial045.stdout:34359738371 2026-02-22T22:58:39.543 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T22:58:39.728 INFO:teuthology.orchestra.run.trial045.stdout:34359738371 2026-02-22T22:58:39.731 INFO:teuthology.orchestra.run.trial045.stdout:34359738370 2026-02-22T22:58:39.738 INFO:teuthology.orchestra.run.trial045.stdout:34359738370 2026-02-22T22:58:39.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2026-02-22T22:58:39.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T22:58:39.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2026-02-22T22:58:40.739 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T22:58:40.743 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T22:58:40.750 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T22:58:40.890 INFO:teuthology.orchestra.run.trial045.stdout:34359738371 2026-02-22T22:58:40.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2026-02-22T22:58:40.928 INFO:teuthology.orchestra.run.trial045.stdout:34359738370 2026-02-22T22:58:40.931 INFO:teuthology.orchestra.run.trial045.stdout:34359738370 2026-02-22T22:58:40.940 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2026-02-22T22:58:40.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2026-02-22T22:58:41.903 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T22:58:41.941 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T22:58:41.944 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T22:58:42.055 INFO:teuthology.orchestra.run.trial045.stdout:34359738373 2026-02-22T22:58:42.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.2 2026-02-22T22:58:42.068 DEBUG:teuthology.parallel:result is None 2026-02-22T22:58:42.099 INFO:teuthology.orchestra.run.trial045.stdout:34359738372 2026-02-22T22:58:42.107 INFO:teuthology.orchestra.run.trial045.stdout:34359738372 2026-02-22T22:58:42.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.1 2026-02-22T22:58:42.110 DEBUG:teuthology.parallel:result is None 2026-02-22T22:58:42.119 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.0 2026-02-22T22:58:42.119 DEBUG:teuthology.parallel:result is None 2026-02-22T22:58:42.119 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T22:58:42.120 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:58:42.306 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:42.307 INFO:teuthology.orchestra.run.trial045.stderr:dumped all 2026-02-22T22:58:42.317 INFO:teuthology.orchestra.run.trial045.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-22T22:58:40.920102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81216,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491584,"statfs":{"total":322122547200,"available":322039382016,"internally_reserved":0,"allocated":327680,"data_stored":81620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944393+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:38.944393+0000","last_peered":"2026-02-22T22:58:38.944393+0000","last_clean":"2026-02-22T22:58:38.944393+0000","last_became_active":"2026-02-22T22:58:36.940675+0000","last_became_peered":"2026-02-22T22:58:36.940675+0000","last_unstale":"2026-02-22T22:58:38.944393+0000","last_undegraded":"2026-02-22T22:58:38.944393+0000","last_fullsized":"2026-02-22T22:58:38.944393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:15:08.584111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058708100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944223+0000","last_change":"2026-02-22T22:58:38.944223+0000","last_active":"2026-02-22T22:58:38.944223+0000","last_peered":"2026-02-22T22:58:38.944223+0000","last_clean":"2026-02-22T22:58:38.944223+0000","last_became_active":"2026-02-22T22:58:36.941017+0000","last_became_peered":"2026-02-22T22:58:36.941017+0000","last_unstale":"2026-02-22T22:58:38.944223+0000","last_undegraded":"2026-02-22T22:58:38.944223+0000","last_fullsized":"2026-02-22T22:58:38.944223+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:59:54.939628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.939984+0000","last_change":"2026-02-22T22:58:38.939984+0000","last_active":"2026-02-22T22:58:38.939984+0000","last_peered":"2026-02-22T22:58:38.939984+0000","last_clean":"2026-02-22T22:58:38.939984+0000","last_became_active":"2026-02-22T22:58:36.940747+0000","last_became_peered":"2026-02-22T22:58:36.940747+0000","last_unstale":"2026-02-22T22:58:38.939984+0000","last_undegraded":"2026-02-22T22:58:38.939984+0000","last_fullsized":"2026-02-22T22:58:38.939984+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:33:54.323398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024186999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944530+0000","last_change":"2026-02-22T22:58:38.944530+0000","last_active":"2026-02-22T22:58:38.944530+0000","last_peered":"2026-02-22T22:58:38.944530+0000","last_clean":"2026-02-22T22:58:38.944530+0000","last_became_active":"2026-02-22T22:58:36.940924+0000","last_became_peered":"2026-02-22T22:58:36.940924+0000","last_unstale":"2026-02-22T22:58:38.944530+0000","last_undegraded":"2026-02-22T22:58:38.944530+0000","last_fullsized":"2026-02-22T22:58:38.944530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:47:11.865384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000718365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":20,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944393+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:38.944393+0000","last_peered":"2026-02-22T22:58:38.944393+0000","last_clean":"2026-02-22T22:58:38.944393+0000","last_became_active":"2026-02-22T22:58:36.940652+0000","last_became_peered":"2026-02-22T22:58:36.940652+0000","last_unstale":"2026-02-22T22:58:38.944393+0000","last_undegraded":"2026-02-22T22:58:38.944393+0000","last_fullsized":"2026-02-22T22:58:38.944393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:09.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.943334+0000","last_change":"2026-02-22T22:58:38.942588+0000","last_active":"2026-02-22T22:58:38.943334+0000","last_peered":"2026-02-22T22:58:38.943334+0000","last_clean":"2026-02-22T22:58:38.943334+0000","last_became_active":"2026-02-22T22:58:36.940457+0000","last_became_peered":"2026-02-22T22:58:36.940457+0000","last_unstale":"2026-02-22T22:58:38.943334+0000","last_undegraded":"2026-02-22T22:58:38.943334+0000","last_fullsized":"2026-02-22T22:58:38.943334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:04:19.266147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048746599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.939974+0000","last_change":"2026-02-22T22:58:38.939974+0000","last_active":"2026-02-22T22:58:38.939974+0000","last_peered":"2026-02-22T22:58:38.939974+0000","last_clean":"2026-02-22T22:58:38.939974+0000","last_became_active":"2026-02-22T22:58:36.940771+0000","last_became_peered":"2026-02-22T22:58:36.940771+0000","last_unstale":"2026-02-22T22:58:38.939974+0000","last_undegraded":"2026-02-22T22:58:38.939974+0000","last_fullsized":"2026-02-22T22:58:38.939974+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:21.807513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944217+0000","last_change":"2026-02-22T22:58:38.944217+0000","last_active":"2026-02-22T22:58:38.944217+0000","last_peered":"2026-02-22T22:58:38.944217+0000","last_clean":"2026-02-22T22:58:38.944217+0000","last_became_active":"2026-02-22T22:58:36.941034+0000","last_became_peered":"2026-02-22T22:58:36.941034+0000","last_unstale":"2026-02-22T22:58:38.944217+0000","last_undegraded":"2026-02-22T22:58:38.944217+0000","last_fullsized":"2026-02-22T22:58:38.944217+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:46:58.470757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046778899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":106496,"data_stored":27194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:58:42.317 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:58:42.460 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:42.460 INFO:teuthology.orchestra.run.trial045.stderr:dumped all 2026-02-22T22:58:42.470 INFO:teuthology.orchestra.run.trial045.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-22T22:58:40.920102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81216,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491584,"statfs":{"total":322122547200,"available":322039382016,"internally_reserved":0,"allocated":327680,"data_stored":81620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944393+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:38.944393+0000","last_peered":"2026-02-22T22:58:38.944393+0000","last_clean":"2026-02-22T22:58:38.944393+0000","last_became_active":"2026-02-22T22:58:36.940675+0000","last_became_peered":"2026-02-22T22:58:36.940675+0000","last_unstale":"2026-02-22T22:58:38.944393+0000","last_undegraded":"2026-02-22T22:58:38.944393+0000","last_fullsized":"2026-02-22T22:58:38.944393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:15:08.584111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058708100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944223+0000","last_change":"2026-02-22T22:58:38.944223+0000","last_active":"2026-02-22T22:58:38.944223+0000","last_peered":"2026-02-22T22:58:38.944223+0000","last_clean":"2026-02-22T22:58:38.944223+0000","last_became_active":"2026-02-22T22:58:36.941017+0000","last_became_peered":"2026-02-22T22:58:36.941017+0000","last_unstale":"2026-02-22T22:58:38.944223+0000","last_undegraded":"2026-02-22T22:58:38.944223+0000","last_fullsized":"2026-02-22T22:58:38.944223+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:59:54.939628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.939984+0000","last_change":"2026-02-22T22:58:38.939984+0000","last_active":"2026-02-22T22:58:38.939984+0000","last_peered":"2026-02-22T22:58:38.939984+0000","last_clean":"2026-02-22T22:58:38.939984+0000","last_became_active":"2026-02-22T22:58:36.940747+0000","last_became_peered":"2026-02-22T22:58:36.940747+0000","last_unstale":"2026-02-22T22:58:38.939984+0000","last_undegraded":"2026-02-22T22:58:38.939984+0000","last_fullsized":"2026-02-22T22:58:38.939984+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:33:54.323398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024186999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944530+0000","last_change":"2026-02-22T22:58:38.944530+0000","last_active":"2026-02-22T22:58:38.944530+0000","last_peered":"2026-02-22T22:58:38.944530+0000","last_clean":"2026-02-22T22:58:38.944530+0000","last_became_active":"2026-02-22T22:58:36.940924+0000","last_became_peered":"2026-02-22T22:58:36.940924+0000","last_unstale":"2026-02-22T22:58:38.944530+0000","last_undegraded":"2026-02-22T22:58:38.944530+0000","last_fullsized":"2026-02-22T22:58:38.944530+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:47:11.865384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000718365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":20,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944393+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:38.944393+0000","last_peered":"2026-02-22T22:58:38.944393+0000","last_clean":"2026-02-22T22:58:38.944393+0000","last_became_active":"2026-02-22T22:58:36.940652+0000","last_became_peered":"2026-02-22T22:58:36.940652+0000","last_unstale":"2026-02-22T22:58:38.944393+0000","last_undegraded":"2026-02-22T22:58:38.944393+0000","last_fullsized":"2026-02-22T22:58:38.944393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:09.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.943334+0000","last_change":"2026-02-22T22:58:38.942588+0000","last_active":"2026-02-22T22:58:38.943334+0000","last_peered":"2026-02-22T22:58:38.943334+0000","last_clean":"2026-02-22T22:58:38.943334+0000","last_became_active":"2026-02-22T22:58:36.940457+0000","last_became_peered":"2026-02-22T22:58:36.940457+0000","last_unstale":"2026-02-22T22:58:38.943334+0000","last_undegraded":"2026-02-22T22:58:38.943334+0000","last_fullsized":"2026-02-22T22:58:38.943334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:04:19.266147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048746599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.939974+0000","last_change":"2026-02-22T22:58:38.939974+0000","last_active":"2026-02-22T22:58:38.939974+0000","last_peered":"2026-02-22T22:58:38.939974+0000","last_clean":"2026-02-22T22:58:38.939974+0000","last_became_active":"2026-02-22T22:58:36.940771+0000","last_became_peered":"2026-02-22T22:58:36.940771+0000","last_unstale":"2026-02-22T22:58:38.939974+0000","last_undegraded":"2026-02-22T22:58:38.939974+0000","last_fullsized":"2026-02-22T22:58:38.939974+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:21.807513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-22T22:58:38.944217+0000","last_change":"2026-02-22T22:58:38.944217+0000","last_active":"2026-02-22T22:58:38.944217+0000","last_peered":"2026-02-22T22:58:38.944217+0000","last_clean":"2026-02-22T22:58:38.944217+0000","last_became_active":"2026-02-22T22:58:36.941034+0000","last_became_peered":"2026-02-22T22:58:36.941034+0000","last_unstale":"2026-02-22T22:58:38.944217+0000","last_undegraded":"2026-02-22T22:58:38.944217+0000","last_fullsized":"2026-02-22T22:58:38.944217+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:46:58.470757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046778899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":106496,"data_stored":27194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:58:42.471 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T22:58:42.472 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-22T22:58:42.472 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-22T22:58:42.472 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T22:58:42.633 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:58:42.633 INFO:teuthology.orchestra.run.trial045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-22T22:58:42.643 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-22T22:58:42.644 INFO:teuthology.run_tasks:Running task workunit... 2026-02-22T22:58:42.671 INFO:tasks.workunit:Pulling workunits from ref f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:58:42.672 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-22T22:58:42.672 INFO:tasks.workunit:timeout=3h 2026-02-22T22:58:42.672 INFO:tasks.workunit:cleanup=True 2026-02-22T22:58:42.672 DEBUG:teuthology.orchestra.run.trial045:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-22T22:58:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:58:42.677 INFO:teuthology.orchestra.run.trial045.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-22T22:58:42.678 DEBUG:teuthology.orchestra.run.trial045:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-22T22:58:42.725 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-22T22:58:42.726 DEBUG:teuthology.orchestra.run.trial045:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-22T22:58:42.773 DEBUG:teuthology.orchestra.run.trial045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:58:42.827 INFO:tasks.workunit.client.0.trial045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-22T22:59:04.856 INFO:tasks.workunit.client.0.trial045.stderr:Note: switching to 'f90f4eaebb28b619cf76e80cbbe997656da69579'. 2026-02-22T22:59:04.856 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:04.856 INFO:tasks.workunit.client.0.trial045.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-22T22:59:04.856 INFO:tasks.workunit.client.0.trial045.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-22T22:59:04.856 INFO:tasks.workunit.client.0.trial045.stderr:state without impacting any branches by switching back to a branch. 2026-02-22T22:59:04.856 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr: git switch -c 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr:Or undo this operation with: 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr: git switch - 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-22T22:59:04.857 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:04.858 INFO:tasks.workunit.client.0.trial045.stderr:HEAD is now at f90f4eaebb2 Merge branch 'wip-74712-squid' of https://github.com/idryomov/ceph into wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:59:04.865 DEBUG:teuthology.orchestra.run.trial045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-22T22:59:04.913 INFO:tasks.workunit.client.0.trial045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-22T22:59:04.915 INFO:tasks.workunit.client.0.trial045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-22T22:59:04.915 INFO:tasks.workunit.client.0.trial045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-22T22:59:05.011 INFO:tasks.workunit.client.0.trial045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-22T22:59:05.042 INFO:tasks.workunit.client.0.trial045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-22T22:59:05.067 INFO:tasks.workunit.client.0.trial045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-22T22:59:05.069 INFO:tasks.workunit.client.0.trial045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-22T22:59:05.069 INFO:tasks.workunit.client.0.trial045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-22T22:59:05.093 INFO:tasks.workunit.client.0.trial045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-22T22:59:05.096 DEBUG:teuthology.orchestra.run.trial045:> set -ex 2026-02-22T22:59:05.096 DEBUG:teuthology.orchestra.run.trial045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-22T22:59:05.141 INFO:tasks.workunit:Running workunits matching cephtool/test_kvstore_tool.sh on client.0... 2026-02-22T22:59:05.144 INFO:tasks.workunit:Running workunit cephtool/test_kvstore_tool.sh... 2026-02-22T22:59:05.144 DEBUG:teuthology.orchestra.run.trial045:workunit test cephtool/test_kvstore_tool.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh 2026-02-22T22:59:05.192 INFO:tasks.workunit.client.0.trial045.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:++ TIMEOUT=300 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:++ MAX_TIMEOUT=15 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:++ PG_NUM=4 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:++ TMPDIR=/tmp 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-22T22:59:05.194 INFO:tasks.workunit.client.0.trial045.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-22T22:59:05.195 INFO:tasks.workunit.client.0.trial045.stderr:++ type xmlstarlet 2026-02-22T22:59:05.195 INFO:tasks.workunit.client.0.trial045.stderr:++ XMLSTARLET=xmlstarlet 2026-02-22T22:59:05.195 INFO:tasks.workunit.client.0.trial045.stderr:+++ uname 2026-02-22T22:59:05.196 INFO:tasks.workunit.client.0.trial045.stderr:++ '[' Linux = FreeBSD ']' 2026-02-22T22:59:05.196 INFO:tasks.workunit.client.0.trial045.stderr:++ SED=sed 2026-02-22T22:59:05.196 INFO:tasks.workunit.client.0.trial045.stderr:++ AWK=awk 2026-02-22T22:59:05.196 INFO:tasks.workunit.client.0.trial045.stderr:+++ stty -a 2026-02-22T22:59:05.196 INFO:tasks.workunit.client.0.trial045.stderr:+++ head -1 2026-02-22T22:59:05.196 INFO:tasks.workunit.client.0.trial045.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-22T22:59:05.197 INFO:tasks.workunit.client.0.trial045.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T22:59:05.197 INFO:tasks.workunit.client.0.trial045.stderr:++ termwidth= 2026-02-22T22:59:05.197 INFO:tasks.workunit.client.0.trial045.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-22T22:59:05.197 INFO:tasks.workunit.client.0.trial045.stderr:++ DIFFCOLOPTS='-y ' 2026-02-22T22:59:05.197 INFO:tasks.workunit.client.0.trial045.stderr:++ KERNCORE=kernel.core_pattern 2026-02-22T22:59:05.197 INFO:tasks.workunit.client.0.trial045.stderr:++ EXTRA_OPTS= 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:++ test '' = TESTS 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:+ set -e 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:+ set -o functrace 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:10: : SUDO=sudo 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:11: : export CEPH_DEV=1 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:11: : CEPH_DEV=1 2026-02-22T22:59:05.200 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:13: : echo note: test ceph_kvstore_tool with bluestore 2026-02-22T22:59:05.201 INFO:tasks.workunit.client.0.trial045.stdout:note: test ceph_kvstore_tool with bluestore 2026-02-22T22:59:05.201 INFO:tasks.workunit.client.0.trial045.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:21: : mktemp -d ./cephtool.XXX 2026-02-22T22:59:05.201 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:21: : TEMP_DIR=./cephtool.WJN 2026-02-22T22:59:05.202 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:22: : trap 'rm -fr ./cephtool.WJN' 0 2026-02-22T22:59:05.202 INFO:tasks.workunit.client.0.trial045.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:24: : mktemp ./cephtool.WJN/test_invalid.XXX 2026-02-22T22:59:05.202 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:24: : TEMP_FILE=./cephtool.WJN/test_invalid.Xci 2026-02-22T22:59:05.203 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:69: : test_ceph_kvstore_tool 2026-02-22T22:59:05.203 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:29: test_ceph_kvstore_tool: ceph-objectstore-tool --data-path ./cephtool.WJN --op mkfs --no-mon-config 2026-02-22T22:59:06.593 INFO:tasks.workunit.client.0.trial045.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:32: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN list 2026-02-22T22:59:06.593 INFO:tasks.workunit.client.0.trial045.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:32: test_ceph_kvstore_tool: wc -l 2026-02-22T22:59:07.674 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:32: test_ceph_kvstore_tool: origin_kv_nums=11 2026-02-22T22:59:07.675 INFO:tasks.workunit.client.0.trial045.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN list 2026-02-22T22:59:07.675 INFO:tasks.workunit.client.0.trial045.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: head -n 1 2026-02-22T22:59:07.675 INFO:tasks.workunit.client.0.trial045.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: awk '{print $1}' 2026-02-22T22:59:08.770 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: prefix=B 2026-02-22T22:59:08.770 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:36: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN exists B 2026-02-22T22:59:08.785 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:08.780+0000 7f61e3d84bc0 1 bdev(0x558fdf4ef800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:08.785 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:08.780+0000 7f61e3d84bc0 1 bdev(0x558fdf4ef800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:08.785 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:08.780+0000 7f61e3d84bc0 1 bdev(0x558fdf4ef800 ./cephtool.WJN/block) close 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bdev(0x558fdf4ef800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bdev(0x558fdf4ef800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bdev(0x558fdf4efc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bdev(0x558fdf4efc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs mount 2026-02-22T22:59:09.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:09.068 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.068 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.068 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.068 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.068 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.068 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:09.068 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.060+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 freelist init 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 freelist _read_cfg 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 0 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000152 seconds, extent_count=1 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:09.073 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bluefs umount 2026-02-22T22:59:09.074 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.068+0000 7f61e3d84bc0 1 bdev(0x558fdf4efc00 ./cephtool.WJN/block) close 2026-02-22T22:59:09.345 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bdev(0x558fdf4efc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:09.345 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bdev(0x558fdf4efc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:09.346 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:09.346 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs mount 2026-02-22T22:59:09.346 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:09.346 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.347 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.347 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.347 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.347 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:09.347 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-22T22:59:09.347 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.340+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:09.354 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.348+0000 7f61e3d84bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:09.354 INFO:tasks.workunit.client.0.trial045.stdout:(B, ) exists 2026-02-22T22:59:09.355 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.348+0000 7f61e3d84bc0 1 bluefs umount 2026-02-22T22:59:09.355 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.348+0000 7f61e3d84bc0 1 bdev(0x558fdf4efc00 ./cephtool.WJN/block) close 2026-02-22T22:59:09.624 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.620+0000 7f61e3d84bc0 1 freelist shutdown 2026-02-22T22:59:09.625 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.620+0000 7f61e3d84bc0 1 bdev(0x558fdf4ef800 ./cephtool.WJN/block) close 2026-02-22T22:59:09.850 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:37: test_ceph_kvstore_tool: expect_false ceph-kvstore-tool bluestore-kv ./cephtool.WJN exists Bnotexist 2026-02-22T22:59:09.850 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:17: expect_false: set -x 2026-02-22T22:59:09.850 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: ceph-kvstore-tool bluestore-kv ./cephtool.WJN exists Bnotexist 2026-02-22T22:59:09.864 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.856+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:09.864 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.856+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:09.865 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:09.860+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25800 ./cephtool.WJN/block) close 2026-02-22T22:59:10.158 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:10.158 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:10.158 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:10.158 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:10.158 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:10.158 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:10.159 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs mount 2026-02-22T22:59:10.159 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:10.160 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.161 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.161 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.161 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.161 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.161 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:10.161 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.152+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:10.166 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:10.166 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:10.167 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:10.167 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:10.167 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:10.167 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 freelist init 2026-02-22T22:59:10.167 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 freelist _read_cfg 2026-02-22T22:59:10.168 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-22T22:59:10.168 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 0 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-22T22:59:10.168 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000167 seconds, extent_count=1 2026-02-22T22:59:10.168 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:10.169 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bluefs umount 2026-02-22T22:59:10.169 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.160+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25c00 ./cephtool.WJN/block) close 2026-02-22T22:59:10.436 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:10.437 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:10.437 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:10.437 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs mount 2026-02-22T22:59:10.437 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:10.438 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.438 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.438 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.438 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.438 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:10.438 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-22T22:59:10.438 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.432+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:10.446 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.440+0000 7f4ae962fbc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:10.446 INFO:tasks.workunit.client.0.trial045.stdout:(Bnotexist, ) does not exist 2026-02-22T22:59:10.446 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.440+0000 7f4ae962fbc0 1 bluefs umount 2026-02-22T22:59:10.446 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.440+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25c00 ./cephtool.WJN/block) close 2026-02-22T22:59:10.717 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.712+0000 7f4ae962fbc0 1 freelist shutdown 2026-02-22T22:59:10.717 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.712+0000 7f4ae962fbc0 1 bdev(0x55d5f7e25800 ./cephtool.WJN/block) close 2026-02-22T22:59:10.941 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: return 0 2026-02-22T22:59:10.941 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:40: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN list-crc 2026-02-22T22:59:10.954 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.948+0000 7f4cb2438bc0 1 bdev(0x5573fca1f800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:10.954 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.948+0000 7f4cb2438bc0 1 bdev(0x5573fca1f800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:10.955 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:10.948+0000 7f4cb2438bc0 1 bdev(0x5573fca1f800 ./cephtool.WJN/block) close 2026-02-22T22:59:11.245 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bdev(0x5573fca1f800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:11.246 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bdev(0x5573fca1f800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:11.246 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:11.246 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bdev(0x5573fca1fc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:11.246 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bdev(0x5573fca1fc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:11.246 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:11.246 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs mount 2026-02-22T22:59:11.246 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:11.248 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.248 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.248 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.248 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.248 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.248 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:11.248 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.240+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:11.254 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:11.254 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:11.254 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 freelist init 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 freelist _read_cfg 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 0 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000111 seconds, extent_count=1 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bluefs umount 2026-02-22T22:59:11.255 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.248+0000 7f4cb2438bc0 1 bdev(0x5573fca1fc00 ./cephtool.WJN/block) close 2026-02-22T22:59:11.525 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bdev(0x5573fca1fc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:11.525 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bdev(0x5573fca1fc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:11.526 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:11.526 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs mount 2026-02-22T22:59:11.526 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:11.526 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.527 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.527 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.527 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.527 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:11.527 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-22T22:59:11.527 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.520+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:11.534 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.528+0000 7f4cb2438bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:11.534 INFO:tasks.workunit.client.0.trial045.stdout:B blocks 3887034139 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:B blocks_per_key 2356384685 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:B bytes_per_block 1745006751 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:B size 714193852 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:S blobid_max 3417961130 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:S freelist_type 753695201 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:S min_alloc_size 1458662255 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:S min_compat_ondisk_format 2796427507 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:S nid_max 1692495819 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:S ondisk_format 3256356326 2026-02-22T22:59:11.535 INFO:tasks.workunit.client.0.trial045.stdout:S per_pool_omap 490065073 2026-02-22T22:59:11.536 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.528+0000 7f4cb2438bc0 1 bluefs umount 2026-02-22T22:59:11.536 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.528+0000 7f4cb2438bc0 1 bdev(0x5573fca1fc00 ./cephtool.WJN/block) close 2026-02-22T22:59:11.813 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.808+0000 7f4cb2438bc0 1 freelist shutdown 2026-02-22T22:59:11.813 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:11.808+0000 7f4cb2438bc0 1 bdev(0x5573fca1f800 ./cephtool.WJN/block) close 2026-02-22T22:59:12.028 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:41: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN list-crc B 2026-02-22T22:59:12.041 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.036+0000 7f42e4183bc0 1 bdev(0x5623e027f800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:12.041 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.036+0000 7f42e4183bc0 1 bdev(0x5623e027f800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:12.041 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.036+0000 7f42e4183bc0 1 bdev(0x5623e027f800 ./cephtool.WJN/block) close 2026-02-22T22:59:12.329 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bdev(0x5623e027f800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:12.329 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bdev(0x5623e027f800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:12.329 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:12.330 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bdev(0x5623e027fc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:12.330 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bdev(0x5623e027fc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:12.330 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:12.330 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs mount 2026-02-22T22:59:12.331 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:12.332 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.332 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.332 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.332 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.332 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.332 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:12.332 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.324+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:12.337 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:12.337 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:12.337 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:12.338 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:12.338 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 freelist init 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 freelist _read_cfg 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 0 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000123 seconds, extent_count=1 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bluefs umount 2026-02-22T22:59:12.339 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.332+0000 7f42e4183bc0 1 bdev(0x5623e027fc00 ./cephtool.WJN/block) close 2026-02-22T22:59:12.609 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bdev(0x5623e027fc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:12.609 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bdev(0x5623e027fc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:12.609 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:12.610 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs mount 2026-02-22T22:59:12.610 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:12.610 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.610 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.610 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.611 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.611 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:12.611 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-22T22:59:12.611 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.604+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:12.618 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.612+0000 7f42e4183bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:12.619 INFO:tasks.workunit.client.0.trial045.stdout:B blocks 3887034139 2026-02-22T22:59:12.619 INFO:tasks.workunit.client.0.trial045.stdout:B blocks_per_key 2356384685 2026-02-22T22:59:12.619 INFO:tasks.workunit.client.0.trial045.stdout:B bytes_per_block 1745006751 2026-02-22T22:59:12.619 INFO:tasks.workunit.client.0.trial045.stdout:B size 714193852 2026-02-22T22:59:12.619 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.612+0000 7f42e4183bc0 1 bluefs umount 2026-02-22T22:59:12.619 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.612+0000 7f42e4183bc0 1 bdev(0x5623e027fc00 ./cephtool.WJN/block) close 2026-02-22T22:59:12.889 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.884+0000 7f42e4183bc0 1 freelist shutdown 2026-02-22T22:59:12.889 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:12.884+0000 7f42e4183bc0 1 bdev(0x5623e027f800 ./cephtool.WJN/block) close 2026-02-22T22:59:13.120 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:44: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN list B 2026-02-22T22:59:13.133 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.128+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:13.133 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.128+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:13.134 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.128+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95800 ./cephtool.WJN/block) close 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs mount 2026-02-22T22:59:13.413 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:13.415 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.415 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.415 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.415 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.415 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.415 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:13.415 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.408+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:13.421 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:13.421 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:13.421 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:13.421 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:13.421 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:13.422 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 freelist init 2026-02-22T22:59:13.422 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 freelist _read_cfg 2026-02-22T22:59:13.422 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-22T22:59:13.422 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 0 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-22T22:59:13.423 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000216 seconds, extent_count=1 2026-02-22T22:59:13.423 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:13.423 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bluefs umount 2026-02-22T22:59:13.423 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.416+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95c00 ./cephtool.WJN/block) close 2026-02-22T22:59:13.701 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:13.701 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:13.702 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:13.702 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs mount 2026-02-22T22:59:13.702 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:13.702 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.702 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.703 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.703 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.703 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:13.703 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-22T22:59:13.703 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.696+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:13.710 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.704+0000 7f2b611a3bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:13.711 INFO:tasks.workunit.client.0.trial045.stdout:B blocks 2026-02-22T22:59:13.711 INFO:tasks.workunit.client.0.trial045.stdout:B blocks_per_key 2026-02-22T22:59:13.711 INFO:tasks.workunit.client.0.trial045.stdout:B bytes_per_block 2026-02-22T22:59:13.711 INFO:tasks.workunit.client.0.trial045.stdout:B size 2026-02-22T22:59:13.711 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.704+0000 7f2b611a3bc0 1 bluefs umount 2026-02-22T22:59:13.712 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.704+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95c00 ./cephtool.WJN/block) close 2026-02-22T22:59:13.985 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.980+0000 7f2b611a3bc0 1 freelist shutdown 2026-02-22T22:59:13.985 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:13.980+0000 7f2b611a3bc0 1 bdev(0x55a7e0c95800 ./cephtool.WJN/block) close 2026-02-22T22:59:14.196 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:47: test_ceph_kvstore_tool: echo helloworld 2026-02-22T22:59:14.196 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:48: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN set TESTPREFIX TESTKEY in ./cephtool.WJN/test_invalid.Xci 2026-02-22T22:59:14.210 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.204+0000 7f4d3d71dbc0 1 bdev(0x55faec64f800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:14.210 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.204+0000 7f4d3d71dbc0 1 bdev(0x55faec64f800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:14.210 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.204+0000 7f4d3d71dbc0 1 bdev(0x55faec64f800 ./cephtool.WJN/block) close 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bdev(0x55faec64f800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bdev(0x55faec64f800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bdev(0x55faec64fc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bdev(0x55faec64fc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs mount 2026-02-22T22:59:14.490 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:14.492 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.492 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.492 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.492 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.492 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.492 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:14.492 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.484+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:14.499 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:14.499 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:14.499 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:14.499 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:14.499 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:14.500 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 freelist init 2026-02-22T22:59:14.500 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 freelist _read_cfg 2026-02-22T22:59:14.501 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-22T22:59:14.501 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 0 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-22T22:59:14.501 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000206 seconds, extent_count=1 2026-02-22T22:59:14.501 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:14.501 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bluefs umount 2026-02-22T22:59:14.501 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.492+0000 7f4d3d71dbc0 1 bdev(0x55faec64fc00 ./cephtool.WJN/block) close 2026-02-22T22:59:14.772 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bdev(0x55faec64fc00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:14.773 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bdev(0x55faec64fc00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:14.773 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:14.774 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs mount 2026-02-22T22:59:14.774 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:14.774 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.774 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.774 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.774 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.774 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:14.775 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-22T22:59:14.775 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.764+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:14.782 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.776+0000 7f4d3d71dbc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:14.785 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.780+0000 7f4d3d71dbc0 1 bluefs umount 2026-02-22T22:59:14.785 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:14.780+0000 7f4d3d71dbc0 1 bdev(0x55faec64fc00 ./cephtool.WJN/block) close 2026-02-22T22:59:15.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.060+0000 7f4d3d71dbc0 1 freelist shutdown 2026-02-22T22:59:15.065 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.060+0000 7f4d3d71dbc0 1 bdev(0x55faec64f800 ./cephtool.WJN/block) close 2026-02-22T22:59:15.272 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:49: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN exists TESTPREFIX TESTKEY 2026-02-22T22:59:15.286 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.280+0000 7f8900ae3bc0 1 bdev(0x55e267a93800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:15.286 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.280+0000 7f8900ae3bc0 1 bdev(0x55e267a93800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:15.287 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.280+0000 7f8900ae3bc0 1 bdev(0x55e267a93800 ./cephtool.WJN/block) close 2026-02-22T22:59:15.569 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bdev(0x55e267a93800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:15.570 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bdev(0x55e267a93800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:15.570 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:15.570 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bdev(0x55e267a93c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:15.570 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bdev(0x55e267a93c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:15.570 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:15.570 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs mount 2026-02-22T22:59:15.571 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:15.572 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.572 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.572 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.572 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.572 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.573 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:15.573 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.564+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:15.580 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.572+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:15.580 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.572+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:15.580 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.572+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:15.580 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.572+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:15.580 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.572+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:15.581 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.572+0000 7f8900ae3bc0 1 freelist init 2026-02-22T22:59:15.581 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.572+0000 7f8900ae3bc0 1 freelist _read_cfg 2026-02-22T22:59:15.581 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.576+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:15.581 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.576+0000 7f8900ae3bc0 1 bluefs umount 2026-02-22T22:59:15.581 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.576+0000 7f8900ae3bc0 1 bdev(0x55e267a93c00 ./cephtool.WJN/block) close 2026-02-22T22:59:15.853 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bdev(0x55e267a93c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:15.853 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bdev(0x55e267a93c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:15.853 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:15.854 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs mount 2026-02-22T22:59:15.854 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:15.854 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.854 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.855 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.855 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.855 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:15.855 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-22T22:59:15.855 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.848+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:15.861 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.856+0000 7f8900ae3bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:15.862 INFO:tasks.workunit.client.0.trial045.stdout:(TESTPREFIX, TESTKEY) exists 2026-02-22T22:59:15.862 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.856+0000 7f8900ae3bc0 1 bluefs umount 2026-02-22T22:59:15.862 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:15.856+0000 7f8900ae3bc0 1 bdev(0x55e267a93c00 ./cephtool.WJN/block) close 2026-02-22T22:59:16.136 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.132+0000 7f8900ae3bc0 1 freelist shutdown 2026-02-22T22:59:16.137 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.132+0000 7f8900ae3bc0 1 bdev(0x55e267a93800 ./cephtool.WJN/block) close 2026-02-22T22:59:16.363 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:52: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN get TESTPREFIX TESTKEY out ./cephtool.WJN/test_invalid.Xci.bak 2026-02-22T22:59:16.377 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.372+0000 7f92753d7bc0 1 bdev(0x55b072d23800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:16.377 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.372+0000 7f92753d7bc0 1 bdev(0x55b072d23800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:16.377 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.372+0000 7f92753d7bc0 1 bdev(0x55b072d23800 ./cephtool.WJN/block) close 2026-02-22T22:59:16.657 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bdev(0x55b072d23800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:16.657 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bdev(0x55b072d23800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:16.658 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:16.658 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bdev(0x55b072d23c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:16.658 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bdev(0x55b072d23c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:16.658 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:16.659 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs mount 2026-02-22T22:59:16.659 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:16.660 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.661 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.661 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.661 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.661 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.661 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:16.661 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.652+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:16.666 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:16.667 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:16.667 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:16.667 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:16.667 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:16.667 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 freelist init 2026-02-22T22:59:16.667 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 freelist _read_cfg 2026-02-22T22:59:16.668 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:16.668 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bluefs umount 2026-02-22T22:59:16.668 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.660+0000 7f92753d7bc0 1 bdev(0x55b072d23c00 ./cephtool.WJN/block) close 2026-02-22T22:59:16.937 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bdev(0x55b072d23c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:16.937 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bdev(0x55b072d23c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:16.937 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:16.938 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs mount 2026-02-22T22:59:16.938 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:16.938 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.938 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.939 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.939 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.939 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:16.939 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-22T22:59:16.939 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.932+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:16.944 INFO:tasks.workunit.client.0.trial045.stdout:(TESTPREFIX, TESTKEY) 2026-02-22T22:59:16.945 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.936+0000 7f92753d7bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:16.945 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.936+0000 7f92753d7bc0 1 bluefs umount 2026-02-22T22:59:16.945 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:16.936+0000 7f92753d7bc0 1 bdev(0x55b072d23c00 ./cephtool.WJN/block) close 2026-02-22T22:59:17.217 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.212+0000 7f92753d7bc0 1 freelist shutdown 2026-02-22T22:59:17.217 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.212+0000 7f92753d7bc0 1 bdev(0x55b072d23800 ./cephtool.WJN/block) close 2026-02-22T22:59:17.441 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:53: test_ceph_kvstore_tool: diff ./cephtool.WJN/test_invalid.Xci ./cephtool.WJN/test_invalid.Xci.bak 2026-02-22T22:59:17.443 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:56: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN rm TESTPREFIX TESTKEY 2026-02-22T22:59:17.464 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.456+0000 7f9e6ec21bc0 1 bdev(0x556d3f641800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:17.464 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.456+0000 7f9e6ec21bc0 1 bdev(0x556d3f641800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:17.464 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.456+0000 7f9e6ec21bc0 1 bdev(0x556d3f641800 ./cephtool.WJN/block) close 2026-02-22T22:59:17.749 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bdev(0x556d3f641800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:17.750 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bdev(0x556d3f641800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:17.750 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:17.750 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bdev(0x556d3f641c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:17.750 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bdev(0x556d3f641c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:17.750 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:17.750 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs mount 2026-02-22T22:59:17.750 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:17.752 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:17.752 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:17.752 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:17.752 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:17.752 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:17.752 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:17.753 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.744+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:17.758 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:17.758 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:17.758 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:17.758 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:17.758 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:17.758 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 freelist init 2026-02-22T22:59:17.759 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 freelist _read_cfg 2026-02-22T22:59:17.759 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:17.759 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bluefs umount 2026-02-22T22:59:17.759 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:17.752+0000 7f9e6ec21bc0 1 bdev(0x556d3f641c00 ./cephtool.WJN/block) close 2026-02-22T22:59:18.029 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bdev(0x556d3f641c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:18.029 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bdev(0x556d3f641c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:18.029 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:18.030 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs mount 2026-02-22T22:59:18.030 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:18.030 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.030 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.030 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.031 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.031 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.031 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-22T22:59:18.031 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.024+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:18.039 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.032+0000 7f9e6ec21bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:18.043 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.036+0000 7f9e6ec21bc0 1 bluefs umount 2026-02-22T22:59:18.043 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.036+0000 7f9e6ec21bc0 1 bdev(0x556d3f641c00 ./cephtool.WJN/block) close 2026-02-22T22:59:18.325 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.320+0000 7f9e6ec21bc0 1 freelist shutdown 2026-02-22T22:59:18.325 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.320+0000 7f9e6ec21bc0 1 bdev(0x556d3f641800 ./cephtool.WJN/block) close 2026-02-22T22:59:18.548 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:57: test_ceph_kvstore_tool: expect_false ceph-kvstore-tool bluestore-kv ./cephtool.WJN exists TESTPREFIX TESTKEY 2026-02-22T22:59:18.548 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:17: expect_false: set -x 2026-02-22T22:59:18.548 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: ceph-kvstore-tool bluestore-kv ./cephtool.WJN exists TESTPREFIX TESTKEY 2026-02-22T22:59:18.561 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.556+0000 7f4f53d74bc0 1 bdev(0x5557353a3800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:18.561 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.556+0000 7f4f53d74bc0 1 bdev(0x5557353a3800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:18.561 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.556+0000 7f4f53d74bc0 1 bdev(0x5557353a3800 ./cephtool.WJN/block) close 2026-02-22T22:59:18.841 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bdev(0x5557353a3800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:18.841 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bdev(0x5557353a3800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:18.842 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:18.842 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bdev(0x5557353a3c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:18.842 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bdev(0x5557353a3c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:18.842 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:18.842 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs mount 2026-02-22T22:59:18.842 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:18.844 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.844 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.844 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.844 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.844 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:18.844 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:18.844 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.836+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:18.850 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:18.850 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:18.850 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:18.850 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:18.850 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:18.850 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 freelist init 2026-02-22T22:59:18.851 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 freelist _read_cfg 2026-02-22T22:59:18.851 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:18.851 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bluefs umount 2026-02-22T22:59:18.852 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:18.844+0000 7f4f53d74bc0 1 bdev(0x5557353a3c00 ./cephtool.WJN/block) close 2026-02-22T22:59:19.120 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bdev(0x5557353a3c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:19.121 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bdev(0x5557353a3c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:19.121 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:19.121 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs mount 2026-02-22T22:59:19.121 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:19.122 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.122 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.122 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.122 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.122 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.122 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluefs mount shared_bdev_used = 4980736 2026-02-22T22:59:19.122 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.116+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:19.128 INFO:tasks.workunit.client.0.trial045.stdout:(TESTPREFIX, TESTKEY) does not exist 2026-02-22T22:59:19.128 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.120+0000 7f4f53d74bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:19.128 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.120+0000 7f4f53d74bc0 1 bluefs umount 2026-02-22T22:59:19.128 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.120+0000 7f4f53d74bc0 1 bdev(0x5557353a3c00 ./cephtool.WJN/block) close 2026-02-22T22:59:19.401 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.396+0000 7f4f53d74bc0 1 freelist shutdown 2026-02-22T22:59:19.401 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.396+0000 7f4f53d74bc0 1 bdev(0x5557353a3800 ./cephtool.WJN/block) close 2026-02-22T22:59:19.632 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: return 0 2026-02-22T22:59:19.632 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:60: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN compact 2026-02-22T22:59:19.646 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.640+0000 7f8165b39bc0 1 bdev(0x55c411b61800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:19.646 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.640+0000 7f8165b39bc0 1 bdev(0x55c411b61800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:19.646 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.640+0000 7f8165b39bc0 1 bdev(0x55c411b61800 ./cephtool.WJN/block) close 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bdev(0x55c411b61800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bdev(0x55c411b61800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bdev(0x55c411b61c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bdev(0x55c411b61c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs mount 2026-02-22T22:59:19.926 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:19.928 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.928 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.928 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.928 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.928 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:19.928 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:19.928 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.920+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:19.934 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:19.934 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:19.934 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:19.934 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:19.934 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:19.934 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 freelist init 2026-02-22T22:59:19.934 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 freelist _read_cfg 2026-02-22T22:59:19.935 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:19.935 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bluefs umount 2026-02-22T22:59:19.935 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:19.928+0000 7f8165b39bc0 1 bdev(0x55c411b61c00 ./cephtool.WJN/block) close 2026-02-22T22:59:20.213 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bdev(0x55c411b61c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:20.213 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bdev(0x55c411b61c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:20.213 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:20.213 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs mount 2026-02-22T22:59:20.214 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:20.214 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:20.214 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:20.214 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:20.214 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:20.215 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:20.215 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluefs mount shared_bdev_used = 4980736 2026-02-22T22:59:20.215 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.208+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:20.223 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.216+0000 7f8165b39bc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:20.224 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.216+0000 7f8165b39bc0 2 rocksdb: compact starting 2026-02-22T22:59:20.227 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.220+0000 7f8165b39bc0 2 rocksdb: compact completed 2026-02-22T22:59:20.229 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.224+0000 7f8165b39bc0 1 bluefs umount 2026-02-22T22:59:20.229 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.224+0000 7f8165b39bc0 1 bdev(0x55c411b61c00 ./cephtool.WJN/block) close 2026-02-22T22:59:20.509 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.504+0000 7f8165b39bc0 1 freelist shutdown 2026-02-22T22:59:20.509 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.504+0000 7f8165b39bc0 1 bdev(0x55c411b61800 ./cephtool.WJN/block) close 2026-02-22T22:59:20.709 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:63: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.WJN destructive-repair 2026-02-22T22:59:20.722 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:20.722 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:20.722 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) close 2026-02-22T22:59:21.009 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.010 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.010 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:21.010 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.010 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.010 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:21.010 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount 2026-02-22T22:59:21.010 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:21.012 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.012 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.013 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.013 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.013 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.013 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:21.013 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:21.017 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.008+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:21.017 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.008+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:21.017 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:21.017 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:21.017 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:21.018 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 freelist init 2026-02-22T22:59:21.018 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 freelist _read_cfg 2026-02-22T22:59:21.018 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:21.019 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluefs umount 2026-02-22T22:59:21.019 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) close 2026-02-22T22:59:21.289 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.289 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.289 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:21.290 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount 2026-02-22T22:59:21.290 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:21.290 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.291 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.291 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.291 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.291 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.291 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-22T22:59:21.291 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:21.313 INFO:tasks.workunit.client.0.trial045.stdout:destructive-repair completed without reporting an error 2026-02-22T22:59:21.318 INFO:tasks.workunit.client.0.trial045.stderr:*** Caught signal (Segmentation fault) ** 2026-02-22T22:59:21.318 INFO:tasks.workunit.client.0.trial045.stderr: in thread 7ffa0d2aebc0 thread_name:ceph-kvstore-to 2026-02-22T22:59:21.318 INFO:tasks.workunit.client.0.trial045.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-22T22:59:21.318 INFO:tasks.workunit.client.0.trial045.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffa0e4b9520] 2026-02-22T22:59:21.318 INFO:tasks.workunit.client.0.trial045.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x55c69f818fa4] 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x55c69f8188d9] 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 4: (BlueStore::_close_db()+0x31c) [0x55c69f66ae9c] 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 5: (BlueStore::umount()+0x239) [0x55c69f675469] 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55c69f616ed3] 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 7: main() 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ffa0e4a0d90] 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 9: __libc_start_main() 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 10: _start() 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr:2026-02-22T22:59:21.308+0000 7ffa0d2aebc0 -1 *** Caught signal (Segmentation fault) ** 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: in thread 7ffa0d2aebc0 thread_name:ceph-kvstore-to 2026-02-22T22:59:21.319 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffa0e4b9520] 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x55c69f818fa4] 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x55c69f8188d9] 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 4: (BlueStore::_close_db()+0x31c) [0x55c69f66ae9c] 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 5: (BlueStore::umount()+0x239) [0x55c69f675469] 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55c69f616ed3] 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 7: main() 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ffa0e4a0d90] 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 9: __libc_start_main() 2026-02-22T22:59:21.320 INFO:tasks.workunit.client.0.trial045.stderr: 10: _start() 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr:--- begin dump of recent events --- 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -190> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command assert hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -189> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command abort hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -188> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command leak_some_memory hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -187> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perfcounters_dump hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -186> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command 1 hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -185> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf dump hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -184> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perfcounters_schema hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -183> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf histogram dump hook 0x55c6b2202280 2026-02-22T22:59:21.321 INFO:tasks.workunit.client.0.trial045.stderr: -182> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command 2 hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -181> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf schema hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -180> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command counter dump hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -179> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command counter schema hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -178> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf histogram schema hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -177> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf reset hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -176> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config show hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -175> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config help hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -174> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config set hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -173> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config unset hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -172> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config get hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -171> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config diff hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -170> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config diff get hook 0x55c6b2202280 2026-02-22T22:59:21.322 INFO:tasks.workunit.client.0.trial045.stderr: -169> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command injectargs hook 0x55c6b2202280 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -168> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command log flush hook 0x55c6b2202280 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -167> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command log dump hook 0x55c6b2202280 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -166> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command log reopen hook 0x55c6b2202280 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -165> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command dump_mempools hook 0x55c6b22ee068 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -164> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _open_db_and_around::NCB::read_only=0, to_repair=1 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -163> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -162> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -161> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) close 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -160> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -159> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -158> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:21.323 INFO:tasks.workunit.client.0.trial045.stderr: -157> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore bluefs device info hook 0x55c6b22022f0 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -156> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs stats hook 0x55c6b22022f0 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -155> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs files list hook 0x55c6b22022f0 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -154> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs debug_inject_read_zeros hook 0x55c6b22022f0 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -153> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -152> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -151> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -150> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -149> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 bluefs mount super: super(uuid a0676ec2-a2b9-4baa-86c6-e1d6cc8f9c43 osd 19bd091e-3165-4b2c-96a4-1b9d88463225 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-22T22:59:20.225767+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -148> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -147> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -146> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -145> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.324 INFO:tasks.workunit.client.0.trial045.stderr: -144> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -143> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -142> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -141> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -140> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -139> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -138> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -137> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -136> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -135> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -134> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.325 INFO:tasks.workunit.client.0.trial045.stderr: -133> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -132> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -131> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -130> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -129> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -128> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -127> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -126> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -125> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -124> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -123> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -122> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -121> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.326 INFO:tasks.workunit.client.0.trial045.stderr: -120> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -119> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -118> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -117> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -116> 2026-02-22T22:59:21.008+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -115> 2026-02-22T22:59:21.008+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -114> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -113> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _open_super_meta::NCB::freelist_type=null 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -112> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -111> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -110> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -109> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor snappy 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -108> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor snappy 0x55c6b256d530 2026-02-22T22:59:21.327 INFO:tasks.workunit.client.0.trial045.stderr: -107> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0x55c6b256d530 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -106> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor snappy loaded and registered 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -105> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0x55c6b256d530 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -104> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zlib = 0 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -103> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor zlib 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -102> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor zlib 0x55c6b256d380 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -101> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zlib = 0x55c6b256d380 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -100> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor zlib loaded and registered 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -99> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zlib = 0x55c6b256d380 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -98> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zstd = 0 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -97> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor zstd 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -96> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor zstd 0x55c6b266c030 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -95> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zstd = 0x55c6b266c030 2026-02-22T22:59:21.328 INFO:tasks.workunit.client.0.trial045.stderr: -94> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor zstd loaded and registered 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -93> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zstd = 0x55c6b266c030 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -92> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor lz4 = 0 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -91> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor lz4 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -90> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor lz4 0x55c6b266c0c0 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -89> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor lz4 = 0x55c6b266c0c0 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -88> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor lz4 loaded and registered 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -87> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor lz4 = 0x55c6b266c0c0 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -86> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0x55c6b256d530 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -85> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _open_fm::NCB::freelist_type=null 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -84> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 freelist init 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -83> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 freelist _read_cfg 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -82> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator dump block hook 0x55c6b266c0f0 2026-02-22T22:59:21.329 INFO:tasks.workunit.client.0.trial045.stderr: -81> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator score block hook 0x55c6b266c0f0 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -80> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation block hook 0x55c6b266c0f0 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -79> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation histogram block hook 0x55c6b266c0f0 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -78> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator dump recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -77> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator score recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -76> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -75> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation histogram recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -74> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::__restore_allocator::file_size=132,sizeof(extent_t)=16 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -73> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::__restore_allocator::READ--extent_count=1, read_alloc_size= 107374174208, file_size=132 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -72> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::__restore_allocator::READ duration=0.000387 seconds, s_serial=3 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -71> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::restore_allocator:: calling copy_allocator(bitmap_allocator -> shared_alloc.a) 2026-02-22T22:59:21.330 INFO:tasks.workunit.client.0.trial045.stderr: -70> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::copy_allocator::count num_entries=1 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -69> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::copy_allocator::copy num_entries=1 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -68> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::restore_allocator::restored in 0.000629 seconds, num_entries=1 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -67> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator dump recovery 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -66> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator fragmentation histogram recovery 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -65> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator fragmentation recovery 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -64> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator score recovery 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -63> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() completed successfully alloc=0x55c6b25f7200 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -62> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -61> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluefs umount 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -60> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluefs debug_inject_read_zeros 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -59> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluefs files list 2026-02-22T22:59:21.331 INFO:tasks.workunit.client.0.trial045.stderr: -58> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluefs stats 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -57> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore bluefs device info 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -56> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) close 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -55> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore bluefs device info hook 0x55c6b2202b90 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -54> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs stats hook 0x55c6b2202b90 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -53> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs files list hook 0x55c6b2202b90 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -52> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs debug_inject_read_zeros hook 0x55c6b2202b90 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -51> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -50> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -49> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -48> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -47> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 bluefs mount super: super(uuid a0676ec2-a2b9-4baa-86c6-e1d6cc8f9c43 osd 19bd091e-3165-4b2c-96a4-1b9d88463225 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-22T22:59:20.225767+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-22T22:59:21.332 INFO:tasks.workunit.client.0.trial045.stderr: -46> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -45> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -44> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -43> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -42> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -41> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -40> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -39> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -38> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -37> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -36> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -35> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.333 INFO:tasks.workunit.client.0.trial045.stderr: -34> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -33> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -32> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -31> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -30> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -29> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -28> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -27> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -26> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 bluestore::NCB::invalidate_allocation_file_on_bluefs::invalidate using bluefs->truncate(p_handle, 0) 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -25> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -24> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -23> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -22> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.334 INFO:tasks.workunit.client.0.trial045.stderr: -21> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -20> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -19> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -18> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -17> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -16> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -15> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -14> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -13> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -12> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -11> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -10> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -9> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.335 INFO:tasks.workunit.client.0.trial045.stderr: -8> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: -7> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: -6> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: -5> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: -4> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: -3> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: -2> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: -1> 2026-02-22T22:59:21.308+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) umount 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: 0> 2026-02-22T22:59:21.308+0000 7ffa0d2aebc0 -1 *** Caught signal (Segmentation fault) ** 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: in thread 7ffa0d2aebc0 thread_name:ceph-kvstore-to 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:21.336 INFO:tasks.workunit.client.0.trial045.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffa0e4b9520] 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x55c69f818fa4] 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x55c69f8188d9] 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 4: (BlueStore::_close_db()+0x31c) [0x55c69f66ae9c] 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 5: (BlueStore::umount()+0x239) [0x55c69f675469] 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55c69f616ed3] 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 7: main() 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ffa0e4a0d90] 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 9: __libc_start_main() 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 10: _start() 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-22T22:59:21.337 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr:--- logging levels --- 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 none 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 lockdep 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 context 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 crush 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_balancer 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_locker 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_log 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_log_expire 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_migrator 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 3/ 5 mds_quiesce 2026-02-22T22:59:21.338 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 buffer 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 timer 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 filer 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 striper 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 objecter 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rados 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd_mirror 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd_replay 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd_pwl 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 journaler 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 objectcacher 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 immutable_obj_cache 2026-02-22T22:59:21.339 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 client 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 osd 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 optracker 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 objclass 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 3 filestore 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 3 journal 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 0 ms 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mon 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 0/10 monc 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 paxos 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 tp 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 auth 2026-02-22T22:59:21.340 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 crypto 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 finisher 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 reserver 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 heartbeatmap 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 perfcounter 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_sync 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_datacache 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_access 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_dbstore 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_flight 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_lifecycle 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_notification 2026-02-22T22:59:21.341 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 javaclient 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 asok 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 throttle 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 0 refs 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 compressor 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 bluestore 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 bluefs 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 3 bdev 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 kstore 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 2/ 2 rocksdb 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 fuse 2026-02-22T22:59:21.342 INFO:tasks.workunit.client.0.trial045.stderr: 2/ 5 mgr 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mgrc 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 dpdk 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 eventtrace 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 prioritycache 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 test 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 cephfs_mirror 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 cephsqlite 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 crimson_interrupt 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_onode 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_odata 2026-02-22T22:59:21.343 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_omap 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_tm 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_t 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_cleaner 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_epm 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_lba 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_fixedkv_tree 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_cache 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_journal 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_device 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_backref 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 alienstore 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mclock 2026-02-22T22:59:21.344 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 cyanstore 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 ceph_exporter 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 memstore 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 trace 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: -2/-2 (syslog threshold) 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: 99/99 (stderr threshold) 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr:--- pthread ID / name mapping for recent threads --- 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: 7ffa0d2aebc0 / ceph-kvstore-to 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: max_recent 500 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: max_new 1000 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: log_file 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr:--- end dump of recent events --- 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr:--- begin dump of recent events --- 2026-02-22T22:59:21.345 INFO:tasks.workunit.client.0.trial045.stderr: -190> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command assert hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -189> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command abort hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -188> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command leak_some_memory hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -187> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perfcounters_dump hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -186> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command 1 hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -185> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf dump hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -184> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perfcounters_schema hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -183> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf histogram dump hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -182> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command 2 hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -181> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf schema hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -180> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command counter dump hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -179> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command counter schema hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -178> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf histogram schema hook 0x55c6b2202280 2026-02-22T22:59:21.346 INFO:tasks.workunit.client.0.trial045.stderr: -177> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command perf reset hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -176> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config show hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -175> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config help hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -174> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config set hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -173> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config unset hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -172> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config get hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -171> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config diff hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -170> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command config diff get hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -169> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command injectargs hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -168> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command log flush hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -167> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command log dump hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -166> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command log reopen hook 0x55c6b2202280 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -165> 2026-02-22T22:59:20.712+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command dump_mempools hook 0x55c6b22ee068 2026-02-22T22:59:21.347 INFO:tasks.workunit.client.0.trial045.stderr: -164> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _open_db_and_around::NCB::read_only=0, to_repair=1 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -163> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -162> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -161> 2026-02-22T22:59:20.716+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) close 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -160> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -159> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229800 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -158> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -157> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore bluefs device info hook 0x55c6b22022f0 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -156> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs stats hook 0x55c6b22022f0 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -155> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs files list hook 0x55c6b22022f0 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -154> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs debug_inject_read_zeros hook 0x55c6b22022f0 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -153> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -152> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.348 INFO:tasks.workunit.client.0.trial045.stderr: -151> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -150> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -149> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 5 bluefs mount super: super(uuid a0676ec2-a2b9-4baa-86c6-e1d6cc8f9c43 osd 19bd091e-3165-4b2c-96a4-1b9d88463225 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-22T22:59:20.225767+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -148> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -147> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -146> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -145> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -144> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -143> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -142> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluefs mount shared_bdev_used = 0 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -141> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -140> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.349 INFO:tasks.workunit.client.0.trial045.stderr: -139> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -138> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -137> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -136> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -135> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -134> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -133> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -132> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -131> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -130> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -129> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -128> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -127> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.350 INFO:tasks.workunit.client.0.trial045.stderr: -126> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -125> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -124> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -123> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -122> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -121> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -120> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -119> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -118> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -117> 2026-02-22T22:59:21.004+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -116> 2026-02-22T22:59:21.008+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -115> 2026-02-22T22:59:21.008+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta old nid_max 0 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -114> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta old blobid_max 0 2026-02-22T22:59:21.351 INFO:tasks.workunit.client.0.trial045.stderr: -113> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _open_super_meta::NCB::freelist_type=null 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -112> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -111> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _open_super_meta min_alloc_size 0x1000 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -110> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -109> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor snappy 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -108> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor snappy 0x55c6b256d530 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -107> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0x55c6b256d530 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -106> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor snappy loaded and registered 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -105> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0x55c6b256d530 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -104> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zlib = 0 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -103> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor zlib 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -102> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor zlib 0x55c6b256d380 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -101> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zlib = 0x55c6b256d380 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -100> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor zlib loaded and registered 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -99> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zlib = 0x55c6b256d380 2026-02-22T22:59:21.352 INFO:tasks.workunit.client.0.trial045.stderr: -98> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zstd = 0 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -97> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor zstd 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -96> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor zstd 0x55c6b266c030 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -95> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zstd = 0x55c6b266c030 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -94> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor zstd loaded and registered 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -93> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor zstd = 0x55c6b266c030 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -92> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor lz4 = 0 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -91> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load compressor lz4 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -90> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 add compressor lz4 0x55c6b266c0c0 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -89> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor lz4 = 0x55c6b266c0c0 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -88> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 load: compressor lz4 loaded and registered 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -87> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor lz4 = 0x55c6b266c0c0 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -86> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 get compressor snappy = 0x55c6b256d530 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -85> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _open_fm::NCB::freelist_type=null 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -84> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 freelist init 2026-02-22T22:59:21.353 INFO:tasks.workunit.client.0.trial045.stderr: -83> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 freelist _read_cfg 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -82> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator dump block hook 0x55c6b266c0f0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -81> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator score block hook 0x55c6b266c0f0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -80> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation block hook 0x55c6b266c0f0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -79> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation histogram block hook 0x55c6b266c0f0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -78> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator dump recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -77> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator score recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -76> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -75> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore allocator fragmentation histogram recovery hook 0x55c6b266c1b0 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -74> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::__restore_allocator::file_size=132,sizeof(extent_t)=16 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -73> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::__restore_allocator::READ--extent_count=1, read_alloc_size= 107374174208, file_size=132 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -72> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::__restore_allocator::READ duration=0.000387 seconds, s_serial=3 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -71> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::restore_allocator:: calling copy_allocator(bitmap_allocator -> shared_alloc.a) 2026-02-22T22:59:21.354 INFO:tasks.workunit.client.0.trial045.stderr: -70> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::copy_allocator::count num_entries=1 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -69> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::copy_allocator::copy num_entries=1 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -68> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore::NCB::restore_allocator::restored in 0.000629 seconds, num_entries=1 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -67> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator dump recovery 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -66> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator fragmentation histogram recovery 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -65> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator fragmentation recovery 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -64> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore allocator score recovery 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -63> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) _init_alloc::NCB::restore_allocator() completed successfully alloc=0x55c6b25f7200 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -62> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -61> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bluefs umount 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -60> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluefs debug_inject_read_zeros 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -59> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluefs files list 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -58> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluefs stats 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -57> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) unregister_commands bluestore bluefs device info 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -56> 2026-02-22T22:59:21.012+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) close 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -55> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluestore bluefs device info hook 0x55c6b2202b90 2026-02-22T22:59:21.355 INFO:tasks.workunit.client.0.trial045.stderr: -54> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs stats hook 0x55c6b2202b90 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -53> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs files list hook 0x55c6b2202b90 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -52> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 asok(0x55c6b2290000) register_command bluefs debug_inject_read_zeros hook 0x55c6b2202b90 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -51> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open path ./cephtool.WJN/block 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -50> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bdev(0x55c6b2229c00 ./cephtool.WJN/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -49> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs add_block_device bdev 1 path ./cephtool.WJN/block size 100 GiB 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -48> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -47> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 bluefs mount super: super(uuid a0676ec2-a2b9-4baa-86c6-e1d6cc8f9c43 osd 19bd091e-3165-4b2c-96a4-1b9d88463225 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-22T22:59:20.225767+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -46> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -45> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -44> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -43> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -42> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -41> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -40> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -39> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 bluestore(./cephtool.WJN) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T22:59:21.356 INFO:tasks.workunit.client.0.trial045.stderr: -38> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -37> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -36> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -35> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -34> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -33> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -32> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -31> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -30> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -29> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -28> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -27> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -26> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 5 bluestore::NCB::invalidate_allocation_file_on_bluefs::invalidate using bluefs->truncate(p_handle, 0) 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -25> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -24> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -23> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -22> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.357 INFO:tasks.workunit.client.0.trial045.stderr: -21> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -20> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -19> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -18> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -17> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -16> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -15> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -14> 2026-02-22T22:59:21.284+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -13> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -12> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -11> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -10> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_background_jobs = 4 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -9> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -8> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -7> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -6> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-22T22:59:21.358 INFO:tasks.workunit.client.0.trial045.stderr: -5> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: -4> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: -3> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: -2> 2026-02-22T22:59:21.288+0000 7ffa0d2aebc0 1 set rocksdb option compression = kLZ4Compression 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: -1> 2026-02-22T22:59:21.308+0000 7ffa0d2aebc0 5 bluestore(./cephtool.WJN) umount 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 0> 2026-02-22T22:59:21.308+0000 7ffa0d2aebc0 -1 *** Caught signal (Segmentation fault) ** 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: in thread 7ffa0d2aebc0 thread_name:ceph-kvstore-to 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ffa0e4b9520] 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x55c69f818fa4] 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x55c69f8188d9] 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 4: (BlueStore::_close_db()+0x31c) [0x55c69f66ae9c] 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 5: (BlueStore::umount()+0x239) [0x55c69f675469] 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55c69f616ed3] 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 7: main() 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ffa0e4a0d90] 2026-02-22T22:59:21.359 INFO:tasks.workunit.client.0.trial045.stderr: 9: __libc_start_main() 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 10: _start() 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr:--- logging levels --- 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 none 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 lockdep 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 context 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 crush 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_balancer 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_locker 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_log 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_log_expire 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mds_migrator 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 3/ 5 mds_quiesce 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 buffer 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 timer 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 filer 2026-02-22T22:59:21.360 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 striper 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 1 objecter 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rados 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd_mirror 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd_replay 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 rbd_pwl 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 journaler 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 objectcacher 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 immutable_obj_cache 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 client 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 osd 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 optracker 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 objclass 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 3 filestore 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 3 journal 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 0 ms 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mon 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 0/10 monc 2026-02-22T22:59:21.361 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 paxos 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 tp 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 auth 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 crypto 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 finisher 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 reserver 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 heartbeatmap 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 perfcounter 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_sync 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_datacache 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_access 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_dbstore 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_flight 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_lifecycle 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 rgw_notification 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 javaclient 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 asok 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 1 throttle 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 0 refs 2026-02-22T22:59:21.362 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 compressor 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 bluestore 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 bluefs 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 3 bdev 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 kstore 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 2/ 2 rocksdb 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 fuse 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 2/ 5 mgr 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mgrc 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 dpdk 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 eventtrace 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 prioritycache 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 test 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 cephfs_mirror 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 cephsqlite 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 crimson_interrupt 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_onode 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_odata 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_omap 2026-02-22T22:59:21.363 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_tm 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_t 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_cleaner 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_epm 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_lba 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_fixedkv_tree 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_cache 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_journal 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_device 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 seastore_backref 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 alienstore 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 mclock 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 0/ 5 cyanstore 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 ceph_exporter 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 memstore 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 1/ 5 trace 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: -2/-2 (syslog threshold) 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 99/99 (stderr threshold) 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr:--- pthread ID / name mapping for recent threads --- 2026-02-22T22:59:21.364 INFO:tasks.workunit.client.0.trial045.stderr: 7ffa0d2aebc0 / ceph-kvstore-to 2026-02-22T22:59:21.365 INFO:tasks.workunit.client.0.trial045.stderr: max_recent 500 2026-02-22T22:59:21.365 INFO:tasks.workunit.client.0.trial045.stderr: max_new 1000 2026-02-22T22:59:21.365 INFO:tasks.workunit.client.0.trial045.stderr: log_file /var/lib/ceph/crash/2026-02-22T22:59:21.311774Z_bbb911e8-42ea-4d7d-9d50-bd93ce7f1311/log 2026-02-22T22:59:21.365 INFO:tasks.workunit.client.0.trial045.stderr:--- end dump of recent events --- 2026-02-22T22:59:21.379 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh: line 27: 22546 Segmentation fault (core dumped) ceph-kvstore-tool bluestore-kv ${TEMP_DIR} destructive-repair 2026-02-22T22:59:21.379 INFO:tasks.workunit.client.0.trial045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:1: test_ceph_kvstore_tool: rm -fr ./cephtool.WJN 2026-02-22T22:59:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 139 2026-02-22T22:59:21.381 INFO:tasks.workunit:Stopping ['cephtool/test_kvstore_tool.sh'] on client.0... 2026-02-22T22:59:21.382 DEBUG:teuthology.orchestra.run.trial045:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-22T22:59:21.695 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/workunit.py", line 144, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/workunit.py", line 292, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/workunit.py", line 433, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial045 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' 2026-02-22T22:59:21.700 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T22:59:21.710 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T22:59:21.710 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:59:21.855 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:59:21.855 INFO:teuthology.orchestra.run.trial045.stderr:dumped all 2026-02-22T22:59:21.865 INFO:teuthology.orchestra.run.trial045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-22T22:59:20.924663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001126"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808903+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808903+0000","last_peered":"2026-02-22T22:58:45.808903+0000","last_clean":"2026-02-22T22:58:45.808903+0000","last_became_active":"2026-02-22T22:58:36.940675+0000","last_became_peered":"2026-02-22T22:58:36.940675+0000","last_unstale":"2026-02-22T22:58:45.808903+0000","last_undegraded":"2026-02-22T22:58:45.808903+0000","last_fullsized":"2026-02-22T22:58:45.808903+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:15:08.584111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058708100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808900+0000","last_change":"2026-02-22T22:58:38.944223+0000","last_active":"2026-02-22T22:58:45.808900+0000","last_peered":"2026-02-22T22:58:45.808900+0000","last_clean":"2026-02-22T22:58:45.808900+0000","last_became_active":"2026-02-22T22:58:36.941017+0000","last_became_peered":"2026-02-22T22:58:36.941017+0000","last_unstale":"2026-02-22T22:58:45.808900+0000","last_undegraded":"2026-02-22T22:58:45.808900+0000","last_fullsized":"2026-02-22T22:58:45.808900+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:59:54.939628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910682+0000","last_change":"2026-02-22T22:58:38.939984+0000","last_active":"2026-02-22T22:58:45.910682+0000","last_peered":"2026-02-22T22:58:45.910682+0000","last_clean":"2026-02-22T22:58:45.910682+0000","last_became_active":"2026-02-22T22:58:36.940747+0000","last_became_peered":"2026-02-22T22:58:36.940747+0000","last_unstale":"2026-02-22T22:58:45.910682+0000","last_undegraded":"2026-02-22T22:58:45.910682+0000","last_fullsized":"2026-02-22T22:58:45.910682+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:33:54.323398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024186999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808878+0000","last_change":"2026-02-22T22:58:38.944530+0000","last_active":"2026-02-22T22:58:45.808878+0000","last_peered":"2026-02-22T22:58:45.808878+0000","last_clean":"2026-02-22T22:58:45.808878+0000","last_became_active":"2026-02-22T22:58:36.940924+0000","last_became_peered":"2026-02-22T22:58:36.940924+0000","last_unstale":"2026-02-22T22:58:45.808878+0000","last_undegraded":"2026-02-22T22:58:45.808878+0000","last_fullsized":"2026-02-22T22:58:45.808878+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:47:11.865384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000718365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808883+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808883+0000","last_peered":"2026-02-22T22:58:45.808883+0000","last_clean":"2026-02-22T22:58:45.808883+0000","last_became_active":"2026-02-22T22:58:36.940652+0000","last_became_peered":"2026-02-22T22:58:36.940652+0000","last_unstale":"2026-02-22T22:58:45.808883+0000","last_undegraded":"2026-02-22T22:58:45.808883+0000","last_fullsized":"2026-02-22T22:58:45.808883+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:09.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.811820+0000","last_change":"2026-02-22T22:58:38.942588+0000","last_active":"2026-02-22T22:58:45.811820+0000","last_peered":"2026-02-22T22:58:45.811820+0000","last_clean":"2026-02-22T22:58:45.811820+0000","last_became_active":"2026-02-22T22:58:36.940457+0000","last_became_peered":"2026-02-22T22:58:36.940457+0000","last_unstale":"2026-02-22T22:58:45.811820+0000","last_undegraded":"2026-02-22T22:58:45.811820+0000","last_fullsized":"2026-02-22T22:58:45.811820+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:04:19.266147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048746599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910676+0000","last_change":"2026-02-22T22:58:38.939974+0000","last_active":"2026-02-22T22:58:45.910676+0000","last_peered":"2026-02-22T22:58:45.910676+0000","last_clean":"2026-02-22T22:58:45.910676+0000","last_became_active":"2026-02-22T22:58:36.940771+0000","last_became_peered":"2026-02-22T22:58:36.940771+0000","last_unstale":"2026-02-22T22:58:45.910676+0000","last_undegraded":"2026-02-22T22:58:45.910676+0000","last_fullsized":"2026-02-22T22:58:45.910676+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:21.807513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808874+0000","last_change":"2026-02-22T22:58:38.944217+0000","last_active":"2026-02-22T22:58:45.808874+0000","last_peered":"2026-02-22T22:58:45.808874+0000","last_clean":"2026-02-22T22:58:45.808874+0000","last_became_active":"2026-02-22T22:58:36.941034+0000","last_became_peered":"2026-02-22T22:58:36.941034+0000","last_unstale":"2026-02-22T22:58:45.808874+0000","last_undegraded":"2026-02-22T22:58:45.808874+0000","last_fullsized":"2026-02-22T22:58:45.808874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:46:58.470757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046778899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:59:21.867 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:59:22.009 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:59:22.009 INFO:teuthology.orchestra.run.trial045.stderr:dumped all 2026-02-22T22:59:22.021 INFO:teuthology.orchestra.run.trial045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-22T22:59:20.924663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001126"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808903+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808903+0000","last_peered":"2026-02-22T22:58:45.808903+0000","last_clean":"2026-02-22T22:58:45.808903+0000","last_became_active":"2026-02-22T22:58:36.940675+0000","last_became_peered":"2026-02-22T22:58:36.940675+0000","last_unstale":"2026-02-22T22:58:45.808903+0000","last_undegraded":"2026-02-22T22:58:45.808903+0000","last_fullsized":"2026-02-22T22:58:45.808903+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:15:08.584111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058708100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808900+0000","last_change":"2026-02-22T22:58:38.944223+0000","last_active":"2026-02-22T22:58:45.808900+0000","last_peered":"2026-02-22T22:58:45.808900+0000","last_clean":"2026-02-22T22:58:45.808900+0000","last_became_active":"2026-02-22T22:58:36.941017+0000","last_became_peered":"2026-02-22T22:58:36.941017+0000","last_unstale":"2026-02-22T22:58:45.808900+0000","last_undegraded":"2026-02-22T22:58:45.808900+0000","last_fullsized":"2026-02-22T22:58:45.808900+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:59:54.939628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910682+0000","last_change":"2026-02-22T22:58:38.939984+0000","last_active":"2026-02-22T22:58:45.910682+0000","last_peered":"2026-02-22T22:58:45.910682+0000","last_clean":"2026-02-22T22:58:45.910682+0000","last_became_active":"2026-02-22T22:58:36.940747+0000","last_became_peered":"2026-02-22T22:58:36.940747+0000","last_unstale":"2026-02-22T22:58:45.910682+0000","last_undegraded":"2026-02-22T22:58:45.910682+0000","last_fullsized":"2026-02-22T22:58:45.910682+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:33:54.323398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024186999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808878+0000","last_change":"2026-02-22T22:58:38.944530+0000","last_active":"2026-02-22T22:58:45.808878+0000","last_peered":"2026-02-22T22:58:45.808878+0000","last_clean":"2026-02-22T22:58:45.808878+0000","last_became_active":"2026-02-22T22:58:36.940924+0000","last_became_peered":"2026-02-22T22:58:36.940924+0000","last_unstale":"2026-02-22T22:58:45.808878+0000","last_undegraded":"2026-02-22T22:58:45.808878+0000","last_fullsized":"2026-02-22T22:58:45.808878+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:47:11.865384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000718365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808883+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808883+0000","last_peered":"2026-02-22T22:58:45.808883+0000","last_clean":"2026-02-22T22:58:45.808883+0000","last_became_active":"2026-02-22T22:58:36.940652+0000","last_became_peered":"2026-02-22T22:58:36.940652+0000","last_unstale":"2026-02-22T22:58:45.808883+0000","last_undegraded":"2026-02-22T22:58:45.808883+0000","last_fullsized":"2026-02-22T22:58:45.808883+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:09.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.811820+0000","last_change":"2026-02-22T22:58:38.942588+0000","last_active":"2026-02-22T22:58:45.811820+0000","last_peered":"2026-02-22T22:58:45.811820+0000","last_clean":"2026-02-22T22:58:45.811820+0000","last_became_active":"2026-02-22T22:58:36.940457+0000","last_became_peered":"2026-02-22T22:58:36.940457+0000","last_unstale":"2026-02-22T22:58:45.811820+0000","last_undegraded":"2026-02-22T22:58:45.811820+0000","last_fullsized":"2026-02-22T22:58:45.811820+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:04:19.266147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048746599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910676+0000","last_change":"2026-02-22T22:58:38.939974+0000","last_active":"2026-02-22T22:58:45.910676+0000","last_peered":"2026-02-22T22:58:45.910676+0000","last_clean":"2026-02-22T22:58:45.910676+0000","last_became_active":"2026-02-22T22:58:36.940771+0000","last_became_peered":"2026-02-22T22:58:36.940771+0000","last_unstale":"2026-02-22T22:58:45.910676+0000","last_undegraded":"2026-02-22T22:58:45.910676+0000","last_fullsized":"2026-02-22T22:58:45.910676+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:21.807513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808874+0000","last_change":"2026-02-22T22:58:38.944217+0000","last_active":"2026-02-22T22:58:45.808874+0000","last_peered":"2026-02-22T22:58:45.808874+0000","last_clean":"2026-02-22T22:58:45.808874+0000","last_became_active":"2026-02-22T22:58:36.941034+0000","last_became_peered":"2026-02-22T22:58:36.941034+0000","last_unstale":"2026-02-22T22:58:45.808874+0000","last_undegraded":"2026-02-22T22:58:45.808874+0000","last_fullsized":"2026-02-22T22:58:45.808874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:46:58.470757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046778899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:59:22.023 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T22:59:22.023 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:59:22.169 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:59:22.170 INFO:teuthology.orchestra.run.trial045.stderr:dumped all 2026-02-22T22:59:22.182 INFO:teuthology.orchestra.run.trial045.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-22T22:59:20.924663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001126"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808903+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808903+0000","last_peered":"2026-02-22T22:58:45.808903+0000","last_clean":"2026-02-22T22:58:45.808903+0000","last_became_active":"2026-02-22T22:58:36.940675+0000","last_became_peered":"2026-02-22T22:58:36.940675+0000","last_unstale":"2026-02-22T22:58:45.808903+0000","last_undegraded":"2026-02-22T22:58:45.808903+0000","last_fullsized":"2026-02-22T22:58:45.808903+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:15:08.584111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058708100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808900+0000","last_change":"2026-02-22T22:58:38.944223+0000","last_active":"2026-02-22T22:58:45.808900+0000","last_peered":"2026-02-22T22:58:45.808900+0000","last_clean":"2026-02-22T22:58:45.808900+0000","last_became_active":"2026-02-22T22:58:36.941017+0000","last_became_peered":"2026-02-22T22:58:36.941017+0000","last_unstale":"2026-02-22T22:58:45.808900+0000","last_undegraded":"2026-02-22T22:58:45.808900+0000","last_fullsized":"2026-02-22T22:58:45.808900+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:59:54.939628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910682+0000","last_change":"2026-02-22T22:58:38.939984+0000","last_active":"2026-02-22T22:58:45.910682+0000","last_peered":"2026-02-22T22:58:45.910682+0000","last_clean":"2026-02-22T22:58:45.910682+0000","last_became_active":"2026-02-22T22:58:36.940747+0000","last_became_peered":"2026-02-22T22:58:36.940747+0000","last_unstale":"2026-02-22T22:58:45.910682+0000","last_undegraded":"2026-02-22T22:58:45.910682+0000","last_fullsized":"2026-02-22T22:58:45.910682+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:33:54.323398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024186999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808878+0000","last_change":"2026-02-22T22:58:38.944530+0000","last_active":"2026-02-22T22:58:45.808878+0000","last_peered":"2026-02-22T22:58:45.808878+0000","last_clean":"2026-02-22T22:58:45.808878+0000","last_became_active":"2026-02-22T22:58:36.940924+0000","last_became_peered":"2026-02-22T22:58:36.940924+0000","last_unstale":"2026-02-22T22:58:45.808878+0000","last_undegraded":"2026-02-22T22:58:45.808878+0000","last_fullsized":"2026-02-22T22:58:45.808878+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:47:11.865384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000718365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808883+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808883+0000","last_peered":"2026-02-22T22:58:45.808883+0000","last_clean":"2026-02-22T22:58:45.808883+0000","last_became_active":"2026-02-22T22:58:36.940652+0000","last_became_peered":"2026-02-22T22:58:36.940652+0000","last_unstale":"2026-02-22T22:58:45.808883+0000","last_undegraded":"2026-02-22T22:58:45.808883+0000","last_fullsized":"2026-02-22T22:58:45.808883+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:09.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.811820+0000","last_change":"2026-02-22T22:58:38.942588+0000","last_active":"2026-02-22T22:58:45.811820+0000","last_peered":"2026-02-22T22:58:45.811820+0000","last_clean":"2026-02-22T22:58:45.811820+0000","last_became_active":"2026-02-22T22:58:36.940457+0000","last_became_peered":"2026-02-22T22:58:36.940457+0000","last_unstale":"2026-02-22T22:58:45.811820+0000","last_undegraded":"2026-02-22T22:58:45.811820+0000","last_fullsized":"2026-02-22T22:58:45.811820+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:04:19.266147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048746599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910676+0000","last_change":"2026-02-22T22:58:38.939974+0000","last_active":"2026-02-22T22:58:45.910676+0000","last_peered":"2026-02-22T22:58:45.910676+0000","last_clean":"2026-02-22T22:58:45.910676+0000","last_became_active":"2026-02-22T22:58:36.940771+0000","last_became_peered":"2026-02-22T22:58:36.940771+0000","last_unstale":"2026-02-22T22:58:45.910676+0000","last_undegraded":"2026-02-22T22:58:45.910676+0000","last_fullsized":"2026-02-22T22:58:45.910676+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:21.807513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808874+0000","last_change":"2026-02-22T22:58:38.944217+0000","last_active":"2026-02-22T22:58:45.808874+0000","last_peered":"2026-02-22T22:58:45.808874+0000","last_clean":"2026-02-22T22:58:45.808874+0000","last_became_active":"2026-02-22T22:58:36.941034+0000","last_became_peered":"2026-02-22T22:58:36.941034+0000","last_unstale":"2026-02-22T22:58:45.808874+0000","last_undegraded":"2026-02-22T22:58:45.808874+0000","last_fullsized":"2026-02-22T22:58:45.808874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:46:58.470757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046778899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:59:22.183 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T22:59:22.326 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:59:22.326 INFO:teuthology.orchestra.run.trial045.stdout:{"epoch":13,"fsid":"63f08d12-d287-4276-9f47-af125eb32eff","created":"2026-02-22T22:58:20.847724+0000","modified":"2026-02-22T22:58:45.807206+0000","last_up_change":"2026-02-22T22:58:28.923362+0000","last_in_change":"2026-02-22T22:58:21.463222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-22T22:58:35.038338+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1f888aa2-0eec-4e5e-8241-a48a22dd208d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6810","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6811","nonce":4207577072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6812","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6813","nonce":4207577072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6816","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6817","nonce":4207577072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6814","nonce":4207577072},{"type":"v1","addr":"10.20.193.45:6815","nonce":4207577072}]},"public_addr":"10.20.193.45:6811/4207577072","cluster_addr":"10.20.193.45:6813/4207577072","heartbeat_back_addr":"10.20.193.45:6817/4207577072","heartbeat_front_addr":"10.20.193.45:6815/4207577072","state":["exists","up"]},{"osd":1,"uuid":"6981b8b6-8c9a-44d8-a43f-0a68ccdb5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6818","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6819","nonce":2231773529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6820","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6821","nonce":2231773529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6824","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6825","nonce":2231773529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6822","nonce":2231773529},{"type":"v1","addr":"10.20.193.45:6823","nonce":2231773529}]},"public_addr":"10.20.193.45:6819/2231773529","cluster_addr":"10.20.193.45:6821/2231773529","heartbeat_back_addr":"10.20.193.45:6825/2231773529","heartbeat_front_addr":"10.20.193.45:6823/2231773529","state":["exists","up"]},{"osd":2,"uuid":"82cd9280-8291-4ad9-a089-6fc67799bcbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6802","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6803","nonce":25824641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6804","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6805","nonce":25824641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6808","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6809","nonce":25824641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.45:6806","nonce":25824641},{"type":"v1","addr":"10.20.193.45:6807","nonce":25824641}]},"public_addr":"10.20.193.45:6803/25824641","cluster_addr":"10.20.193.45:6805/25824641","heartbeat_back_addr":"10.20.193.45:6809/25824641","heartbeat_front_addr":"10.20.193.45:6807/25824641","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T22:59:23.339 INFO:tasks.ceph:Scrubbing osd.0 2026-02-22T22:59:23.339 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T22:59:23.423 INFO:teuthology.orchestra.run.trial045.stdout:{ 2026-02-22T22:59:23.423 INFO:teuthology.orchestra.run.trial045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T22:59:23.423 INFO:teuthology.orchestra.run.trial045.stdout:} 2026-02-22T22:59:23.432 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-22T22:59:23.573 INFO:teuthology.orchestra.run.trial045.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T22:59:23.584 INFO:tasks.ceph:Scrubbing osd.1 2026-02-22T22:59:23.585 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T22:59:23.657 INFO:teuthology.orchestra.run.trial045.stdout:{ 2026-02-22T22:59:23.657 INFO:teuthology.orchestra.run.trial045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T22:59:23.657 INFO:teuthology.orchestra.run.trial045.stdout:} 2026-02-22T22:59:23.666 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-22T22:59:23.814 INFO:teuthology.orchestra.run.trial045.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T22:59:23.825 INFO:tasks.ceph:Scrubbing osd.2 2026-02-22T22:59:23.825 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T22:59:23.897 INFO:teuthology.orchestra.run.trial045.stdout:{ 2026-02-22T22:59:23.897 INFO:teuthology.orchestra.run.trial045.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T22:59:23.897 INFO:teuthology.orchestra.run.trial045.stdout:} 2026-02-22T22:59:23.906 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-22T22:59:24.054 INFO:teuthology.orchestra.run.trial045.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T22:59:24.066 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:59:24.214 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:59:24.214 INFO:teuthology.orchestra.run.trial045.stderr:dumped all 2026-02-22T22:59:24.224 INFO:teuthology.orchestra.run.trial045.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2026-02-22T22:59:22.924872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001080"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808903+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808903+0000","last_peered":"2026-02-22T22:58:45.808903+0000","last_clean":"2026-02-22T22:58:45.808903+0000","last_became_active":"2026-02-22T22:58:36.940675+0000","last_became_peered":"2026-02-22T22:58:36.940675+0000","last_unstale":"2026-02-22T22:58:45.808903+0000","last_undegraded":"2026-02-22T22:58:45.808903+0000","last_fullsized":"2026-02-22T22:58:45.808903+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:15:08.584111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058708100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808900+0000","last_change":"2026-02-22T22:58:38.944223+0000","last_active":"2026-02-22T22:58:45.808900+0000","last_peered":"2026-02-22T22:58:45.808900+0000","last_clean":"2026-02-22T22:58:45.808900+0000","last_became_active":"2026-02-22T22:58:36.941017+0000","last_became_peered":"2026-02-22T22:58:36.941017+0000","last_unstale":"2026-02-22T22:58:45.808900+0000","last_undegraded":"2026-02-22T22:58:45.808900+0000","last_fullsized":"2026-02-22T22:58:45.808900+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:59:54.939628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910682+0000","last_change":"2026-02-22T22:58:38.939984+0000","last_active":"2026-02-22T22:58:45.910682+0000","last_peered":"2026-02-22T22:58:45.910682+0000","last_clean":"2026-02-22T22:58:45.910682+0000","last_became_active":"2026-02-22T22:58:36.940747+0000","last_became_peered":"2026-02-22T22:58:36.940747+0000","last_unstale":"2026-02-22T22:58:45.910682+0000","last_undegraded":"2026-02-22T22:58:45.910682+0000","last_fullsized":"2026-02-22T22:58:45.910682+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:33:54.323398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024186999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808878+0000","last_change":"2026-02-22T22:58:38.944530+0000","last_active":"2026-02-22T22:58:45.808878+0000","last_peered":"2026-02-22T22:58:45.808878+0000","last_clean":"2026-02-22T22:58:45.808878+0000","last_became_active":"2026-02-22T22:58:36.940924+0000","last_became_peered":"2026-02-22T22:58:36.940924+0000","last_unstale":"2026-02-22T22:58:45.808878+0000","last_undegraded":"2026-02-22T22:58:45.808878+0000","last_fullsized":"2026-02-22T22:58:45.808878+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:47:11.865384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000718365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808883+0000","last_change":"2026-02-22T22:58:38.944393+0000","last_active":"2026-02-22T22:58:45.808883+0000","last_peered":"2026-02-22T22:58:45.808883+0000","last_clean":"2026-02-22T22:58:45.808883+0000","last_became_active":"2026-02-22T22:58:36.940652+0000","last_became_peered":"2026-02-22T22:58:36.940652+0000","last_unstale":"2026-02-22T22:58:45.808883+0000","last_undegraded":"2026-02-22T22:58:45.808883+0000","last_fullsized":"2026-02-22T22:58:45.808883+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:21:09.578015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.811820+0000","last_change":"2026-02-22T22:58:38.942588+0000","last_active":"2026-02-22T22:58:45.811820+0000","last_peered":"2026-02-22T22:58:45.811820+0000","last_clean":"2026-02-22T22:58:45.811820+0000","last_became_active":"2026-02-22T22:58:36.940457+0000","last_became_peered":"2026-02-22T22:58:36.940457+0000","last_unstale":"2026-02-22T22:58:45.811820+0000","last_undegraded":"2026-02-22T22:58:45.811820+0000","last_fullsized":"2026-02-22T22:58:45.811820+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:04:19.266147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048746599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.910676+0000","last_change":"2026-02-22T22:58:38.939974+0000","last_active":"2026-02-22T22:58:45.910676+0000","last_peered":"2026-02-22T22:58:45.910676+0000","last_clean":"2026-02-22T22:58:45.910676+0000","last_became_active":"2026-02-22T22:58:36.940771+0000","last_became_peered":"2026-02-22T22:58:36.940771+0000","last_unstale":"2026-02-22T22:58:45.910676+0000","last_undegraded":"2026-02-22T22:58:45.910676+0000","last_fullsized":"2026-02-22T22:58:45.910676+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:39:21.807513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:58:45.808874+0000","last_change":"2026-02-22T22:58:38.944217+0000","last_active":"2026-02-22T22:58:45.808874+0000","last_peered":"2026-02-22T22:58:45.808874+0000","last_clean":"2026-02-22T22:58:45.808874+0000","last_became_active":"2026-02-22T22:58:36.941034+0000","last_became_peered":"2026-02-22T22:58:36.941034+0000","last_unstale":"2026-02-22T22:58:45.808874+0000","last_undegraded":"2026-02-22T22:58:45.808874+0000","last_fullsized":"2026-02-22T22:58:45.808874+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:58:35.931881+0000","last_clean_scrub_stamp":"2026-02-22T22:58:35.931881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:46:58.470757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046778899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738380,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:59:24.229 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.229 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.230 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.230 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.230 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.230 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.230 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.230 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T22:58:35.931881+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=58, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=22, tm_min=59, tm_sec=22, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T22:59:24.231 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T22:59:44.231 DEBUG:teuthology.orchestra.run.trial045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T22:59:44.387 INFO:teuthology.orchestra.run.trial045.stdout: 2026-02-22T22:59:44.388 INFO:teuthology.orchestra.run.trial045.stderr:dumped all 2026-02-22T22:59:44.398 INFO:teuthology.orchestra.run.trial045.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2026-02-22T22:59:43.996997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81224,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491576,"statfs":{"total":322122547200,"available":322039373824,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82767186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.071038"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:23.859014+0000","last_change":"2026-02-22T22:59:23.859030+0000","last_active":"2026-02-22T22:59:23.859014+0000","last_peered":"2026-02-22T22:59:23.859014+0000","last_clean":"2026-02-22T22:59:23.859014+0000","last_became_active":"2026-02-22T22:58:36.940675+0000","last_became_peered":"2026-02-22T22:58:36.940675+0000","last_unstale":"2026-02-22T22:59:23.859014+0000","last_undegraded":"2026-02-22T22:59:23.859014+0000","last_fullsized":"2026-02-22T22:59:23.859014+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:59:23.858958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:59:23.858958+0000","last_clean_scrub_stamp":"2026-02-22T22:59:23.858958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:59:59.960766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058708100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:25.904568+0000","last_change":"2026-02-22T22:59:25.904583+0000","last_active":"2026-02-22T22:59:25.904568+0000","last_peered":"2026-02-22T22:59:25.904568+0000","last_clean":"2026-02-22T22:59:25.904568+0000","last_became_active":"2026-02-22T22:58:36.941017+0000","last_became_peered":"2026-02-22T22:58:36.941017+0000","last_unstale":"2026-02-22T22:59:25.904568+0000","last_undegraded":"2026-02-22T22:59:25.904568+0000","last_fullsized":"2026-02-22T22:59:25.904568+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:59:25.904524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:59:25.904524+0000","last_clean_scrub_stamp":"2026-02-22T22:59:25.904524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:24:48.119999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045315200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:24.994531+0000","last_change":"2026-02-22T22:59:24.994547+0000","last_active":"2026-02-22T22:59:24.994531+0000","last_peered":"2026-02-22T22:59:24.994531+0000","last_clean":"2026-02-22T22:59:24.994531+0000","last_became_active":"2026-02-22T22:58:36.940747+0000","last_became_peered":"2026-02-22T22:58:36.940747+0000","last_unstale":"2026-02-22T22:59:24.994531+0000","last_undegraded":"2026-02-22T22:59:24.994531+0000","last_fullsized":"2026-02-22T22:59:24.994531+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:59:24.994482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:59:24.994482+0000","last_clean_scrub_stamp":"2026-02-22T22:59:24.994482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:15:16.280567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024186999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:24.861746+0000","last_change":"2026-02-22T22:59:24.861766+0000","last_active":"2026-02-22T22:59:24.861746+0000","last_peered":"2026-02-22T22:59:24.861746+0000","last_clean":"2026-02-22T22:59:24.861746+0000","last_became_active":"2026-02-22T22:58:36.940924+0000","last_became_peered":"2026-02-22T22:58:36.940924+0000","last_unstale":"2026-02-22T22:59:24.861746+0000","last_undegraded":"2026-02-22T22:59:24.861746+0000","last_fullsized":"2026-02-22T22:59:24.861746+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:59:24.861694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:59:24.861694+0000","last_clean_scrub_stamp":"2026-02-22T22:59:24.861694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:52:36.057198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000718365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'2","reported_seq":32,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:27.921733+0000","last_change":"2026-02-22T22:59:27.921756+0000","last_active":"2026-02-22T22:59:27.921733+0000","last_peered":"2026-02-22T22:59:27.921733+0000","last_clean":"2026-02-22T22:59:27.921733+0000","last_became_active":"2026-02-22T22:58:36.940652+0000","last_became_peered":"2026-02-22T22:58:36.940652+0000","last_unstale":"2026-02-22T22:59:27.921733+0000","last_undegraded":"2026-02-22T22:59:27.921733+0000","last_fullsized":"2026-02-22T22:59:27.921733+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'2","last_scrub_stamp":"2026-02-22T22:59:27.921681+0000","last_deep_scrub":"13'2","last_deep_scrub_stamp":"2026-02-22T22:59:27.921681+0000","last_clean_scrub_stamp":"2026-02-22T22:59:27.921681+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:03:37.312913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061478300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'3","reported_seq":33,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:23.939870+0000","last_change":"2026-02-22T22:59:23.939888+0000","last_active":"2026-02-22T22:59:23.939870+0000","last_peered":"2026-02-22T22:59:23.939870+0000","last_clean":"2026-02-22T22:59:23.939870+0000","last_became_active":"2026-02-22T22:58:36.940457+0000","last_became_peered":"2026-02-22T22:58:36.940457+0000","last_unstale":"2026-02-22T22:59:23.939870+0000","last_undegraded":"2026-02-22T22:59:23.939870+0000","last_fullsized":"2026-02-22T22:59:23.939870+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'3","last_scrub_stamp":"2026-02-22T22:59:23.939837+0000","last_deep_scrub":"13'3","last_deep_scrub_stamp":"2026-02-22T22:59:23.939837+0000","last_clean_scrub_stamp":"2026-02-22T22:59:23.939837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:21:42.242285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00048746599999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:26.033968+0000","last_change":"2026-02-22T22:59:26.033990+0000","last_active":"2026-02-22T22:59:26.033968+0000","last_peered":"2026-02-22T22:59:26.033968+0000","last_clean":"2026-02-22T22:59:26.033968+0000","last_became_active":"2026-02-22T22:58:36.940771+0000","last_became_peered":"2026-02-22T22:58:36.940771+0000","last_unstale":"2026-02-22T22:59:26.033968+0000","last_undegraded":"2026-02-22T22:59:26.033968+0000","last_fullsized":"2026-02-22T22:59:26.033968+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:59:26.033910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:59:26.033910+0000","last_clean_scrub_stamp":"2026-02-22T22:59:26.033910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T00:09:21.090337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000238283,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T22:59:26.915124+0000","last_change":"2026-02-22T22:59:26.915139+0000","last_active":"2026-02-22T22:59:26.915124+0000","last_peered":"2026-02-22T22:59:26.915124+0000","last_clean":"2026-02-22T22:59:26.915124+0000","last_became_active":"2026-02-22T22:58:36.941034+0000","last_became_peered":"2026-02-22T22:58:36.941034+0000","last_unstale":"2026-02-22T22:59:26.915124+0000","last_undegraded":"2026-02-22T22:59:26.915124+0000","last_fullsized":"2026-02-22T22:59:26.915124+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T22:59:26.915068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T22:59:26.915068+0000","last_clean_scrub_stamp":"2026-02-22T22:59:26.915068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:05:14.945810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00046778899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738385,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738384,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738384,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T22:59:44.400 DEBUG:teuthology.orchestra.run.trial045:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-22T22:59:44.559 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T22:59:44.559 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T22:59:44.560 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-22T22:59:44.560 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:59:44.632 INFO:tasks.ceph.osd.0:Stopped 2026-02-22T22:59:44.633 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-22T22:59:44.633 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:59:44.720 INFO:tasks.ceph.osd.1:Stopped 2026-02-22T22:59:44.721 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-22T22:59:44.721 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:59:45.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-22T22:59:50.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2026-02-22T22:59:50.822 INFO:tasks.ceph.osd.2:Stopped 2026-02-22T22:59:50.822 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T22:59:50.823 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-22T22:59:50.823 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:59:50.852 INFO:tasks.ceph.mgr.x:Stopped 2026-02-22T22:59:50.853 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T22:59:50.853 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-22T22:59:50.853 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T22:59:50.908 INFO:tasks.ceph.mon.a:Stopped 2026-02-22T22:59:50.909 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T22:59:50.909 DEBUG:teuthology.orchestra.run.trial045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-22T22:59:50.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:59:50.963 DEBUG:teuthology.orchestra.run.trial045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-22T22:59:51.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:59:51.016 DEBUG:teuthology.orchestra.run.trial045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-22T22:59:51.072 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:59:51.072 DEBUG:teuthology.orchestra.run.trial045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-22T22:59:51.123 INFO:tasks.ceph:Archiving mon data... 2026-02-22T22:59:51.125 DEBUG:teuthology.misc:Transferring archived files from trial045:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311/data/mon.a.tgz 2026-02-22T22:59:51.125 DEBUG:teuthology.orchestra.run.trial045:> mktemp 2026-02-22T22:59:51.168 INFO:teuthology.orchestra.run.trial045.stdout:/tmp/tmp.TrKbImCSz1 2026-02-22T22:59:51.168 DEBUG:teuthology.orchestra.run.trial045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TrKbImCSz1 2026-02-22T22:59:51.233 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod 0666 /tmp/tmp.TrKbImCSz1 2026-02-22T22:59:51.330 DEBUG:teuthology.orchestra.remote:trial045:/tmp/tmp.TrKbImCSz1 is 130KB 2026-02-22T22:59:51.342 DEBUG:teuthology.orchestra.run.trial045:> rm -fr /tmp/tmp.TrKbImCSz1 2026-02-22T22:59:51.347 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T22:59:51.347 DEBUG:teuthology.orchestra.run.trial045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T22:59:51.440 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T22:59:51.440 INFO:tasks.ceph:Archiving crash dumps... 2026-02-22T22:59:51.442 DEBUG:teuthology.misc:Transferring archived files from trial045:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311/remote/trial045/crash 2026-02-22T22:59:51.443 DEBUG:teuthology.orchestra.run.trial045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T22:59:51.488 INFO:tasks.ceph:Compressing logs... 2026-02-22T22:59:51.489 DEBUG:teuthology.orchestra.run.trial045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T22:59:51.540 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22826.log 2026-02-22T22:59:51.544 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16953.log 2026-02-22T22:59:51.544 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.22826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16847.log 2026-02-22T22:59:51.544 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22826.log.gz 2026-02-22T22:59:51.544 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22982.log 2026-02-22T22:59:51.544 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16953.log.gz 2026-02-22T22:59:51.544 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16847.log.gz 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17107.log 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.22982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16795.log 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22982.log.gz 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.15247.log 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17107.log.gz 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22774.log 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16795.log.gz 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.15247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16957.log 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.15247.log.gz 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.22774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.15047.log 2026-02-22T22:59:51.545 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22774.log.gz 2026-02-22T22:59:51.546 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22930.log 2026-02-22T22:59:51.546 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16957.log.gz 2026-02-22T22:59:51.546 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.15047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17185.log 2026-02-22T22:59:51.546 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.15047.log.gz 2026-02-22T22:59:51.555 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.22930.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22930.log.gz 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23060.log 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17185.log.gz 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23034.log 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.23060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17037.log 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23060.log.gz 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.15184.log 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.23034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23034.log.gz 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17037.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-22T22:59:51.556 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17037.log.gz 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.15184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.15184.log.gz 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.15101.log 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16879.log 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.15101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16724.log 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.15101.log.gz 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16700.log 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16879.log.gz 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23084.log 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16724.log.gz 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16700.log.gz 2026-02-22T22:59:51.557 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.15125.log 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22904.log 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.23084.log: /var/log/ceph/ceph-client.admin.15125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23084.log.gz 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.15125.log.gz 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17237.log 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.22904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22852.log 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22904.log.gz 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.15149.log 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17237.log: /var/log/ceph/ceph-client.admin.22852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17237.log.gz 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22852.log.gz 2026-02-22T22:59:51.558 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23008.log 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.15149.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.15149.log.gz 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.23008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22800.log 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23008.log.gz 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.22800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22800.log.gz 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17115.log 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16881.log 2026-02-22T22:59:51.559 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17115.log: 83.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17115.log.gz 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16749.log 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17211.log 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16881.log.gz 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22956.log 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16749.log.gz 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17211.log.gz 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16774.log 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.22956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22956.log.gz 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16821.log 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16774.log: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-22T22:59:51.560 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16774.log.gz 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17036.log 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16821.log.gz 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16880.log 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12500.log 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17036.log.gz 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16880.log.gz 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22878.log 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph.tmp-client.admin.12500.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12500.log.gz 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17112.log 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.22878.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-22T22:59:51.561 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22878.log.gz 2026-02-22T22:59:51.562 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16959.log 2026-02-22T22:59:51.562 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17112.log.gz 2026-02-22T22:59:51.562 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17029.log 2026-02-22T22:59:51.563 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.16959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14981.log 2026-02-22T22:59:51.563 INFO:teuthology.orchestra.run.trial045.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16959.log.gz 2026-02-22T22:59:51.563 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.17029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17029.log.gz 2026-02-22T22:59:51.564 INFO:teuthology.orchestra.run.trial045.stderr:/var/log/ceph/ceph-client.admin.14981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.14981.log.gz 2026-02-22T22:59:51.566 INFO:teuthology.orchestra.run.trial045.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-22T22:59:51.571 INFO:teuthology.orchestra.run.trial045.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-22T22:59:51.574 INFO:teuthology.orchestra.run.trial045.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-22T22:59:51.574 INFO:teuthology.orchestra.run.trial045.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-22T22:59:51.575 INFO:teuthology.orchestra.run.trial045.stderr: 2026-02-22T22:59:51.575 INFO:teuthology.orchestra.run.trial045.stderr:real 0m0.041s 2026-02-22T22:59:51.575 INFO:teuthology.orchestra.run.trial045.stderr:user 0m0.130s 2026-02-22T22:59:51.575 INFO:teuthology.orchestra.run.trial045.stderr:sys 0m0.009s 2026-02-22T22:59:51.576 INFO:tasks.ceph:Archiving logs... 2026-02-22T22:59:51.576 DEBUG:teuthology.misc:Transferring archived files from trial045:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311/remote/trial045/log 2026-02-22T22:59:51.578 DEBUG:teuthology.orchestra.run.trial045:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T22:59:51.705 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T22:59:51.728 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T22:59:51.728 DEBUG:teuthology.orchestra.run.trial045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T22:59:51.839 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T22:59:51.852 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T22:59:51.852 DEBUG:teuthology.orchestra.run.trial045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T22:59:51.871 INFO:teuthology.orchestra.run.trial045.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T22:59:51.871 INFO:teuthology.orchestra.run.trial045.stdout:============================================================================== 2026-02-22T22:59:51.871 INFO:teuthology.orchestra.run.trial045.stdout: LAX.CALTICK.NET 17.253.83.251 2 u 60 64 3 66.615 +1.663 2.208 2026-02-22T22:59:51.871 INFO:teuthology.orchestra.run.trial045.stdout: 139-177-202-26. 82.43.52.28 2 u 60 64 3 23.246 +9.804 0.728 2026-02-22T22:59:51.871 INFO:teuthology.orchestra.run.trial045.stdout: 45.83.234.123 ( 10.1.105.4 2 u 58 64 3 81.635 +2.020 5.482 2026-02-22T22:59:51.871 INFO:teuthology.orchestra.run.trial045.stdout: 44.190.5.123 17.253.16.253 2 u 54 64 3 66.650 -0.397 0.362 2026-02-22T22:59:51.871 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T22:59:51.882 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T22:59:51.882 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T22:59:51.896 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T22:59:51.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T22:59:51.924 INFO:teuthology.task.internal:Duration was 279.968706 seconds 2026-02-22T22:59:51.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T22:59:51.940 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T22:59:51.940 DEBUG:teuthology.orchestra.run.trial045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T22:59:51.964 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T22:59:51.964 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial045.front.sepia.ceph.com 2026-02-22T22:59:51.964 DEBUG:teuthology.orchestra.run.trial045:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T22:59:52.010 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T22:59:52.010 DEBUG:teuthology.orchestra.run.trial045:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T22:59:52.266 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T22:59:52.267 DEBUG:teuthology.orchestra.run.trial045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T22:59:52.275 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T22:59:52.275 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T22:59:52.275 INFO:teuthology.orchestra.run.trial045.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T22:59:52.276 INFO:teuthology.orchestra.run.trial045.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T22:59:52.276 INFO:teuthology.orchestra.run.trial045.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T22:59:52.343 INFO:teuthology.orchestra.run.trial045.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T22:59:52.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T22:59:52.363 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T22:59:52.363 DEBUG:teuthology.orchestra.run.trial045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T22:59:52.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T22:59:52.413 DEBUG:teuthology.orchestra.run.trial045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:59:52.447 INFO:teuthology.orchestra.run.trial045.stdout:kernel.core_pattern = core 2026-02-22T22:59:52.459 DEBUG:teuthology.orchestra.run.trial045:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:59:52.503 WARNING:teuthology.task.internal:Found coredumps on ubuntu@trial045.front.sepia.ceph.com, flagging run as failed 2026-02-22T22:59:52.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T22:59:52.529 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T22:59:52.531 DEBUG:teuthology.misc:Transferring archived files from trial045:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311/remote/trial045 2026-02-22T22:59:52.531 DEBUG:teuthology.orchestra.run.trial045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T22:59:54.982 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2026-02-22T22:59:54.986 INFO:teuthology.task.internal: core looks like: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64311/remote/trial045/coredump/1771801161.22546.core.gz: gzip compressed data, was "1771801161.22546.core", last modified: Sun Feb 22 22:59:52 2026, max compression, original size modulo 2^32 129236992 2026-02-22T22:59:54.986 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2026-02-22T22:59:55.157 INFO:teuthology.task.internal: core looks like: /tmp/tmpunsn1xb8: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-kvstore-tool bluestore-kv ./cephtool.WJN destructive-repair', real uid: 1000, effective uid: 1000, real gid: 1002, effective gid: 1002, execfn: '/usr/bin/ceph-kvstore-tool', platform: 'x86_64' 2026-02-22T22:59:55.159 INFO:teuthology.task.internal: dump_program: ceph-kvstore-tool 2026-02-22T22:59:55.160 DEBUG:teuthology.orchestra.run.trial045:> which ceph-kvstore-tool 2026-02-22T22:59:55.164 INFO:teuthology.orchestra.run.trial045.stdout:/usr/bin/ceph-kvstore-tool 2026-02-22T22:59:55.210 DEBUG:teuthology.orchestra.remote:trial045:/usr/bin/ceph-kvstore-tool is 10MB 2026-02-22T22:59:55.358 DEBUG:teuthology.orchestra.run.trial045:> mktemp 2026-02-22T22:59:55.361 INFO:teuthology.orchestra.run.trial045.stdout:/tmp/tmp.AB1LyTuN1p 2026-02-22T22:59:55.361 DEBUG:teuthology.orchestra.run.trial045:> sudo cp /usr/bin/ceph-kvstore-tool /tmp/tmp.AB1LyTuN1p 2026-02-22T22:59:55.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-22T22:59:55.426 DEBUG:teuthology.orchestra.run.trial045:> sudo chmod 0666 /tmp/tmp.AB1LyTuN1p 2026-02-22T22:59:55.520 DEBUG:teuthology.orchestra.remote:trial045:/tmp/tmp.AB1LyTuN1p is 10MB 2026-02-22T22:59:55.641 DEBUG:teuthology.orchestra.run.trial045:> rm -fr /tmp/tmp.AB1LyTuN1p 2026-02-22T22:59:55.648 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T22:59:55.648 DEBUG:teuthology.orchestra.run.trial045:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T22:59:55.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T22:59:55.717 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T22:59:55.717 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T22:59:55.733 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T22:59:55.733 DEBUG:teuthology.orchestra.run.trial045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T22:59:55.748 INFO:teuthology.orchestra.run.trial045.stdout: 1179677 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 22 22:59 /home/ubuntu/cephtest 2026-02-22T22:59:55.748 INFO:teuthology.orchestra.run.trial045.stdout: 1189334 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 22 22:58 /home/ubuntu/cephtest/mnt.0 2026-02-22T22:59:55.748 INFO:teuthology.orchestra.run.trial045.stdout: 1189335 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 22 22:59 /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T22:59:55.749 INFO:teuthology.orchestra.run.trial045.stdout: 1204084 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 22 22:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T22:59:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:59:55.750 INFO:teuthology.orchestra.run.trial045.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2026-02-22T22:59:55.750 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial045 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-22T22:59:55.752 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T22:59:55.769 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T22:59:55.786 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on trial045 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-22T22:59:55.792 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} duration: 279.9687056541443 failure_reason: 'Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial045 with status 139: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh''' flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-22T22:59:55.793 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T22:59:55.911 INFO:teuthology.run:FAIL