2026-02-24T18:13:07.641 INFO:root:teuthology version: 1.2.4.dev11+ge6d9099ce 2026-02-24T18:13:07.656 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T18:13:07.739 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} email: null first_in_suite: false flavor: default job_id: '68642' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - overall HEALTH_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5907 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial020.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFP9n/0BnzkUCBW+t12y0FKb2phILCpQ7wUwUFW3B67xkJt8UuwtRzvs6e7IaLPAk1pCfqK63RccEY0WiPzfZbg= tasks: - install: null - ceph: log-ignorelist: - \(POOL_APP_NOT_ENABLED\) - workunit: clients: all: - cephtool/test_kvstore_tool.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: reimage-ssh-down teuthology_sha1: e6d9099ce03dc21193fda653159ec82465c3a94a timestamp: 2026-02-24_16:22:43 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-24T18:13:07.739 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-24T18:13:07.740 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-24T18:13:07.740 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-24T18:13:07.742 INFO:teuthology.task.internal:Checking packages... 2026-02-24T18:13:07.742 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-24T18:13:07.742 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-24T18:13:07.742 INFO:teuthology.packaging:ref: None 2026-02-24T18:13:07.742 INFO:teuthology.packaging:tag: None 2026-02-24T18:13:07.742 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-24T18:13:07.742 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-24T18:13:07.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-24T18:13:07.865 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689-gf90f4eae-1jammy 2026-02-24T18:13:07.866 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-24T18:13:07.879 INFO:teuthology.task.internal:no buildpackages task found 2026-02-24T18:13:07.879 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-24T18:13:07.893 INFO:teuthology.task.internal:Saving configuration 2026-02-24T18:13:07.909 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-24T18:13:07.921 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-24T18:13:07.996 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial020.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 18:09:38.404947', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIInmsV2OpWgDVNMH5V8M4WCPmY3E/BSo7aJ19abS5Rxb'} 2026-02-24T18:13:07.997 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-24T18:13:08.008 INFO:teuthology.task.internal:roles: ubuntu@trial020.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-24T18:13:08.008 INFO:teuthology.run_tasks:Running task console_log... 2026-02-24T18:13:08.092 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7a5fc30453a0>, signals=[15]) 2026-02-24T18:13:08.092 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-24T18:13:08.104 INFO:teuthology.task.internal:Opening connections... 2026-02-24T18:13:08.104 DEBUG:teuthology.task.internal:connecting to ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:13:08.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T18:13:08.189 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-24T18:13:08.201 DEBUG:teuthology.orchestra.run.trial020:> uname -m 2026-02-24T18:13:08.267 INFO:teuthology.orchestra.run.trial020.stdout:x86_64 2026-02-24T18:13:08.268 DEBUG:teuthology.orchestra.run.trial020:> cat /etc/os-release 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:NAME="Ubuntu" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:VERSION_ID="22.04" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:VERSION_CODENAME=jammy 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:ID=ubuntu 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:ID_LIKE=debian 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-24T18:13:08.312 INFO:teuthology.orchestra.run.trial020.stdout:UBUNTU_CODENAME=jammy 2026-02-24T18:13:08.312 INFO:teuthology.lock.ops:Updating trial020.front.sepia.ceph.com on lock server 2026-02-24T18:13:08.378 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-24T18:13:08.390 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-24T18:13:08.402 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-24T18:13:08.402 DEBUG:teuthology.orchestra.run.trial020:> test '!' -e /home/ubuntu/cephtest 2026-02-24T18:13:08.405 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-24T18:13:08.416 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-24T18:13:08.416 DEBUG:teuthology.orchestra.run.trial020:> test -z $(ls -A /var/lib/ceph) 2026-02-24T18:13:08.456 INFO:teuthology.orchestra.run.trial020.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-24T18:13:08.456 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-24T18:13:08.546 INFO:teuthology.run_tasks:Running task kernel... 2026-02-24T18:13:08.566 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-24T18:13:08.566 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-24T18:13:08.566 DEBUG:teuthology.orchestra.run.trial020:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T18:13:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:13:08.570 DEBUG:teuthology.orchestra.run.trial020:> uname -r 2026-02-24T18:13:08.620 INFO:teuthology.orchestra.run.trial020.stdout:5.15.0-170-generic 2026-02-24T18:13:08.620 INFO:teuthology.task.kernel:Running kernel on trial020: 5.15.0-170-generic 2026-02-24T18:13:08.620 DEBUG:teuthology.orchestra.run.trial020:> sudo apt-get clean 2026-02-24T18:13:08.680 DEBUG:teuthology.orchestra.run.trial020:> sudo apt-get update 2026-02-24T18:13:09.197 INFO:teuthology.orchestra.run.trial020.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-24T18:13:09.286 INFO:teuthology.orchestra.run.trial020.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-24T18:13:09.620 INFO:teuthology.orchestra.run.trial020.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-24T18:13:09.769 INFO:teuthology.orchestra.run.trial020.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-24T18:13:09.909 INFO:teuthology.orchestra.run.trial020.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3269 kB] 2026-02-24T18:13:10.309 INFO:teuthology.orchestra.run.trial020.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [497 kB] 2026-02-24T18:13:10.321 INFO:teuthology.orchestra.run.trial020.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB] 2026-02-24T18:13:10.321 INFO:teuthology.orchestra.run.trial020.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-24T18:13:10.356 INFO:teuthology.orchestra.run.trial020.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-24T18:13:10.771 INFO:teuthology.orchestra.run.trial020.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.4 kB] 2026-02-24T18:13:10.985 INFO:teuthology.orchestra.run.trial020.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5379 kB] 2026-02-24T18:13:11.614 INFO:teuthology.orchestra.run.trial020.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1022 kB] 2026-02-24T18:13:11.638 INFO:teuthology.orchestra.run.trial020.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3005 kB] 2026-02-24T18:13:11.723 INFO:teuthology.orchestra.run.trial020.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [429 kB] 2026-02-24T18:13:11.739 INFO:teuthology.orchestra.run.trial020.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-24T18:13:11.739 INFO:teuthology.orchestra.run.trial020.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-24T18:13:11.771 INFO:teuthology.orchestra.run.trial020.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB] 2026-02-24T18:13:11.775 INFO:teuthology.orchestra.run.trial020.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-24T18:13:11.777 INFO:teuthology.orchestra.run.trial020.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5204 kB] 2026-02-24T18:13:11.924 INFO:teuthology.orchestra.run.trial020.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [990 kB] 2026-02-24T18:13:15.314 INFO:teuthology.orchestra.run.trial020.stdout:Fetched 23.1 MB in 4s (5871 kB/s) 2026-02-24T18:13:15.855 INFO:teuthology.orchestra.run.trial020.stdout:Reading package lists... 2026-02-24T18:13:15.870 DEBUG:teuthology.orchestra.run.trial020:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-24T18:13:15.903 INFO:teuthology.orchestra.run.trial020.stdout:Reading package lists... 2026-02-24T18:13:16.013 INFO:teuthology.orchestra.run.trial020.stdout:Building dependency tree... 2026-02-24T18:13:16.014 INFO:teuthology.orchestra.run.trial020.stdout:Reading state information... 2026-02-24T18:13:16.105 INFO:teuthology.orchestra.run.trial020.stdout:The following packages were automatically installed and are no longer required: 2026-02-24T18:13:16.105 INFO:teuthology.orchestra.run.trial020.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout: sg3-utils-udev 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout:The following additional packages will be installed: 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout: linux-generic linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout: linux-headers-generic linux-image-5.15.0-171-generic 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout: linux-modules-5.15.0-171-generic linux-modules-extra-5.15.0-171-generic 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout:Suggested packages: 2026-02-24T18:13:16.106 INFO:teuthology.orchestra.run.trial020.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-24T18:13:16.119 INFO:teuthology.orchestra.run.trial020.stdout:The following NEW packages will be installed: 2026-02-24T18:13:16.119 INFO:teuthology.orchestra.run.trial020.stdout: linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-24T18:13:16.120 INFO:teuthology.orchestra.run.trial020.stdout: linux-image-5.15.0-171-generic linux-modules-5.15.0-171-generic 2026-02-24T18:13:16.120 INFO:teuthology.orchestra.run.trial020.stdout: linux-modules-extra-5.15.0-171-generic 2026-02-24T18:13:16.120 INFO:teuthology.orchestra.run.trial020.stdout:The following packages will be upgraded: 2026-02-24T18:13:16.120 INFO:teuthology.orchestra.run.trial020.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-24T18:13:16.464 INFO:teuthology.orchestra.run.trial020.stdout:3 upgraded, 5 newly installed, 0 to remove and 14 not upgraded. 2026-02-24T18:13:16.464 INFO:teuthology.orchestra.run.trial020.stdout:Need to get 113 MB of archives. 2026-02-24T18:13:16.464 INFO:teuthology.orchestra.run.trial020.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-24T18:13:16.464 INFO:teuthology.orchestra.run.trial020.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-171-generic amd64 5.15.0-171.181 [22.7 MB] 2026-02-24T18:13:17.556 INFO:teuthology.orchestra.run.trial020.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-171-generic amd64 5.15.0-171.181 [11.6 MB] 2026-02-24T18:13:17.825 INFO:teuthology.orchestra.run.trial020.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-171-generic amd64 5.15.0-171.181 [63.9 MB] 2026-02-24T18:13:19.314 INFO:teuthology.orchestra.run.trial020.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.171.160 [1658 B] 2026-02-24T18:13:19.314 INFO:teuthology.orchestra.run.trial020.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.171.160 [2486 B] 2026-02-24T18:13:19.314 INFO:teuthology.orchestra.run.trial020.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171 all 5.15.0-171.181 [12.4 MB] 2026-02-24T18:13:19.590 INFO:teuthology.orchestra.run.trial020.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171-generic amd64 5.15.0-171.181 [2849 kB] 2026-02-24T18:13:19.655 INFO:teuthology.orchestra.run.trial020.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.171.160 [2332 B] 2026-02-24T18:13:19.854 INFO:teuthology.orchestra.run.trial020.stdout:Fetched 113 MB in 4s (32.1 MB/s) 2026-02-24T18:13:19.874 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package linux-modules-5.15.0-171-generic. 2026-02-24T18:13:20.048 INFO:teuthology.orchestra.run.trial020.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-24T18:13:20.050 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../0-linux-modules-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T18:13:20.051 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:20.405 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package linux-image-5.15.0-171-generic. 2026-02-24T18:13:20.413 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../1-linux-image-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T18:13:20.434 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:20.476 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-171-generic. 2026-02-24T18:13:20.484 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T18:13:20.485 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:21.585 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../3-linux-generic_5.15.0.171.160_amd64.deb ... 2026-02-24T18:13:21.586 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-24T18:13:21.613 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.171.160_amd64.deb ... 2026-02-24T18:13:21.614 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-image-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-24T18:13:21.631 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package linux-headers-5.15.0-171. 2026-02-24T18:13:21.639 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../5-linux-headers-5.15.0-171_5.15.0-171.181_all.deb ... 2026-02-24T18:13:21.639 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-24T18:13:23.018 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package linux-headers-5.15.0-171-generic. 2026-02-24T18:13:23.032 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../6-linux-headers-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-24T18:13:23.033 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:23.618 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.171.160_amd64.deb ... 2026-02-24T18:13:23.619 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking linux-headers-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-24T18:13:23.645 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-24T18:13:23.646 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:23.648 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-headers-generic (5.15.0.171.160) ... 2026-02-24T18:13:23.650 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:24.423 INFO:teuthology.orchestra.run.trial020.stdout:I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-24T18:13:24.423 INFO:teuthology.orchestra.run.trial020.stdout:I: /boot/initrd.img.old is now a symlink to initrd.img-5.15.0-170-generic 2026-02-24T18:13:24.424 INFO:teuthology.orchestra.run.trial020.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-171-generic 2026-02-24T18:13:24.424 INFO:teuthology.orchestra.run.trial020.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-171-generic 2026-02-24T18:13:24.432 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:25.230 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:26.018 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-image-generic (5.15.0.171.160) ... 2026-02-24T18:13:26.019 INFO:teuthology.orchestra.run.trial020.stdout:Setting up linux-generic (5.15.0.171.160) ... 2026-02-24T18:13:26.020 INFO:teuthology.orchestra.run.trial020.stdout:Processing triggers for linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-24T18:13:26.025 INFO:teuthology.orchestra.run.trial020.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-24T18:13:26.025 INFO:teuthology.orchestra.run.trial020.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-171-generic 2026-02-24T18:13:33.093 INFO:teuthology.orchestra.run.trial020.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-24T18:13:39.751 INFO:teuthology.orchestra.run.trial020.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-24T18:13:39.751 INFO:teuthology.orchestra.run.trial020.stdout:Sourcing file `/etc/default/grub' 2026-02-24T18:13:39.770 INFO:teuthology.orchestra.run.trial020.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-24T18:13:39.771 INFO:teuthology.orchestra.run.trial020.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-24T18:13:39.773 INFO:teuthology.orchestra.run.trial020.stdout:Generating grub configuration file ... 2026-02-24T18:13:39.856 INFO:teuthology.orchestra.run.trial020.stdout:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-24T18:13:39.863 INFO:teuthology.orchestra.run.trial020.stdout:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-24T18:13:40.038 INFO:teuthology.orchestra.run.trial020.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-24T18:13:40.043 INFO:teuthology.orchestra.run.trial020.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-24T18:13:40.086 INFO:teuthology.orchestra.run.trial020.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-24T18:13:40.092 INFO:teuthology.orchestra.run.trial020.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-24T18:13:40.169 INFO:teuthology.orchestra.run.trial020.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-24T18:13:40.169 INFO:teuthology.orchestra.run.trial020.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-24T18:13:40.170 INFO:teuthology.orchestra.run.trial020.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-24T18:13:40.172 INFO:teuthology.orchestra.run.trial020.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T18:13:40.180 INFO:teuthology.orchestra.run.trial020.stdout:done 2026-02-24T18:13:40.575 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.575 INFO:teuthology.orchestra.run.trial020.stdout:Pending kernel upgrade! 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout:Running kernel version: 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout: 5.15.0-170-generic 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout:Diagnostics: 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout: The currently running kernel version is not the expected kernel version 5.15.0-171-generic. 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout:The processor microcode seems to be up-to-date. 2026-02-24T18:13:40.576 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.577 INFO:teuthology.orchestra.run.trial020.stdout:No services need to be restarted. 2026-02-24T18:13:40.577 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.577 INFO:teuthology.orchestra.run.trial020.stdout:No containers need to be restarted. 2026-02-24T18:13:40.577 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.577 INFO:teuthology.orchestra.run.trial020.stdout:No user sessions are running outdated binaries. 2026-02-24T18:13:40.577 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:13:40.577 INFO:teuthology.orchestra.run.trial020.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-24T18:13:41.380 DEBUG:teuthology.orchestra.run.trial020:> dpkg -s linux-image-generic 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Package: linux-image-generic 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Status: install ok installed 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Priority: optional 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Section: kernel 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Installed-Size: 22 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Maintainer: Ubuntu Kernel Team 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Architecture: amd64 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Source: linux-meta 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Version: 5.15.0.171.160 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Recommends: thermald 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout:Description: Generic Linux kernel image 2026-02-24T18:13:41.389 INFO:teuthology.orchestra.run.trial020.stdout: This package will always depend on the latest generic kernel image 2026-02-24T18:13:41.390 INFO:teuthology.orchestra.run.trial020.stdout: available. 2026-02-24T18:13:41.390 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-170-generic Expected: 5.15.0-171-generic 2026-02-24T18:13:41.390 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T18:13:41.390 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T18:13:41.390 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-24T18:13:41.390 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial020.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T18:13:41.390 DEBUG:teuthology.orchestra.run.trial020:> sudo apt-get clean 2026-02-24T18:13:41.472 DEBUG:teuthology.orchestra.run.trial020:> sudo apt-get update 2026-02-24T18:13:41.962 INFO:teuthology.orchestra.run.trial020.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-24T18:13:42.044 INFO:teuthology.orchestra.run.trial020.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-24T18:13:42.132 INFO:teuthology.orchestra.run.trial020.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-24T18:13:42.220 INFO:teuthology.orchestra.run.trial020.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-24T18:13:42.877 INFO:teuthology.orchestra.run.trial020.stdout:Reading package lists... 2026-02-24T18:13:42.892 DEBUG:teuthology.orchestra.run.trial020:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-24T18:13:42.925 INFO:teuthology.orchestra.run.trial020.stdout:Reading package lists... 2026-02-24T18:13:43.033 INFO:teuthology.orchestra.run.trial020.stdout:Building dependency tree... 2026-02-24T18:13:43.033 INFO:teuthology.orchestra.run.trial020.stdout:Reading state information... 2026-02-24T18:13:43.120 INFO:teuthology.orchestra.run.trial020.stdout:linux-image-generic is already the newest version (5.15.0.171.160). 2026-02-24T18:13:43.120 INFO:teuthology.orchestra.run.trial020.stdout:linux-image-generic set to manually installed. 2026-02-24T18:13:43.121 INFO:teuthology.orchestra.run.trial020.stdout:The following packages were automatically installed and are no longer required: 2026-02-24T18:13:43.121 INFO:teuthology.orchestra.run.trial020.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-24T18:13:43.121 INFO:teuthology.orchestra.run.trial020.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-24T18:13:43.121 INFO:teuthology.orchestra.run.trial020.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-24T18:13:43.121 INFO:teuthology.orchestra.run.trial020.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-24T18:13:43.121 INFO:teuthology.orchestra.run.trial020.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-24T18:13:43.121 INFO:teuthology.orchestra.run.trial020.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-24T18:13:43.140 INFO:teuthology.orchestra.run.trial020.stdout:0 upgraded, 0 newly installed, 0 to remove and 14 not upgraded. 2026-02-24T18:13:43.142 DEBUG:teuthology.orchestra.run.trial020:> dpkg -s linux-image-generic 2026-02-24T18:13:43.193 INFO:teuthology.orchestra.run.trial020.stdout:Package: linux-image-generic 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Status: install ok installed 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Priority: optional 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Section: kernel 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Installed-Size: 22 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Maintainer: Ubuntu Kernel Team 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Architecture: amd64 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Source: linux-meta 2026-02-24T18:13:43.194 INFO:teuthology.orchestra.run.trial020.stdout:Version: 5.15.0.171.160 2026-02-24T18:13:43.195 INFO:teuthology.orchestra.run.trial020.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-24T18:13:43.195 INFO:teuthology.orchestra.run.trial020.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-24T18:13:43.195 INFO:teuthology.orchestra.run.trial020.stdout:Recommends: thermald 2026-02-24T18:13:43.195 INFO:teuthology.orchestra.run.trial020.stdout:Description: Generic Linux kernel image 2026-02-24T18:13:43.195 INFO:teuthology.orchestra.run.trial020.stdout: This package will always depend on the latest generic kernel image 2026-02-24T18:13:43.195 INFO:teuthology.orchestra.run.trial020.stdout: available. 2026-02-24T18:13:43.195 DEBUG:teuthology.orchestra.run.trial020:> mktemp 2026-02-24T18:13:43.241 INFO:teuthology.orchestra.run.trial020.stdout:/tmp/tmp.XyJZ3HBjLV 2026-02-24T18:13:43.241 DEBUG:teuthology.orchestra.run.trial020:> sudo cp /boot/grub/grub.cfg /tmp/tmp.XyJZ3HBjLV 2026-02-24T18:13:43.293 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0666 /tmp/tmp.XyJZ3HBjLV 2026-02-24T18:13:43.394 DEBUG:teuthology.orchestra.remote:trial020:/tmp/tmp.XyJZ3HBjLV is 9KB 2026-02-24T18:13:43.403 DEBUG:teuthology.orchestra.run.trial020:> rm -fr /tmp/tmp.XyJZ3HBjLV 2026-02-24T18:13:43.407 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-24T18:13:43.460 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:13:43.460 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-24T18:13:43.460 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-24T18:13:43.513 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-171-generic 2026-02-24T18:13:43.513 DEBUG:teuthology.orchestra.run.trial020:> sudo update-grub 2026-02-24T18:13:43.749 INFO:teuthology.orchestra.run.trial020.stderr:Sourcing file `/etc/default/grub' 2026-02-24T18:13:43.770 INFO:teuthology.orchestra.run.trial020.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-24T18:13:43.771 INFO:teuthology.orchestra.run.trial020.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-24T18:13:43.773 INFO:teuthology.orchestra.run.trial020.stderr:Generating grub configuration file ... 2026-02-24T18:13:43.854 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-24T18:13:43.860 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-24T18:13:44.036 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-24T18:13:44.041 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-24T18:13:44.084 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-24T18:13:44.089 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-24T18:13:44.161 INFO:teuthology.orchestra.run.trial020.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-24T18:13:44.162 INFO:teuthology.orchestra.run.trial020.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-24T18:13:44.162 INFO:teuthology.orchestra.run.trial020.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-24T18:13:44.163 INFO:teuthology.orchestra.run.trial020.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T18:13:44.171 INFO:teuthology.orchestra.run.trial020.stderr:done 2026-02-24T18:13:44.173 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial020 2026-02-24T18:13:44.173 DEBUG:teuthology.orchestra.run.trial020:> sudo bash -c ' 2026-02-24T18:13:44.173 DEBUG:teuthology.orchestra.run.trial020:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T18:13:44.173 DEBUG:teuthology.orchestra.run.trial020:> ' 2026-02-24T18:14:14.176 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-24T18:14:14.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T18:14:14.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T18:14:28.480 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.20 2026-02-24T18:14:37.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T18:14:37.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T18:15:13.669 DEBUG:teuthology.orchestra.run.trial020:> true 2026-02-24T18:15:14.062 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T18:15:14.062 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-171-generic"... 2026-02-24T18:15:14.062 DEBUG:teuthology.orchestra.run.trial020:> uname -r 2026-02-24T18:15:14.111 INFO:teuthology.orchestra.run.trial020.stdout:5.15.0-171-generic 2026-02-24T18:15:14.111 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-171-generic vs 5.15.0-171-generic 2026-02-24T18:15:14.111 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T18:15:14.112 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-24T18:15:15.113 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-24T18:15:15.114 DEBUG:teuthology.orchestra.run.trial020:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T18:15:15.127 INFO:teuthology.orchestra.run.trial020.stdout:ttyS1 2026-02-24T18:15:15.157 DEBUG:teuthology.parallel:result is None 2026-02-24T18:15:15.157 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-24T18:15:15.170 INFO:teuthology.task.internal:Creating test directory... 2026-02-24T18:15:15.171 DEBUG:teuthology.orchestra.run.trial020:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T18:15:15.203 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-24T18:15:15.214 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-24T18:15:15.226 INFO:teuthology.task.internal:Creating archive directory... 2026-02-24T18:15:15.227 DEBUG:teuthology.orchestra.run.trial020:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T18:15:15.257 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-24T18:15:15.268 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-24T18:15:15.268 DEBUG:teuthology.orchestra.run.trial020:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T18:15:15.303 INFO:teuthology.orchestra.run.trial020.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T18:15:15.309 INFO:teuthology.orchestra.run.trial020.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T18:15:15.311 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-24T18:15:15.323 INFO:teuthology.task.internal:Configuring sudo... 2026-02-24T18:15:15.323 DEBUG:teuthology.orchestra.run.trial020:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T18:15:15.363 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-24T18:15:15.380 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-24T18:15:15.381 DEBUG:teuthology.orchestra.run.trial020:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T18:15:15.414 DEBUG:teuthology.orchestra.run.trial020:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T18:15:15.458 DEBUG:teuthology.orchestra.run.trial020:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T18:15:15.503 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:15:15.503 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T18:15:15.555 DEBUG:teuthology.orchestra.run.trial020:> sudo service rsyslog restart 2026-02-24T18:15:15.616 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-24T18:15:15.629 INFO:teuthology.task.internal:Starting timer... 2026-02-24T18:15:15.630 INFO:teuthology.run_tasks:Running task pcp... 2026-02-24T18:15:15.651 INFO:teuthology.run_tasks:Running task selinux... 2026-02-24T18:15:15.670 DEBUG:teuthology.task.selinux:Excluding trial020: OS 'ubuntu' does not support SELinux 2026-02-24T18:15:15.670 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-24T18:15:15.670 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-24T18:15:15.670 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-24T18:15:15.671 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-24T18:15:15.686 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-24T18:15:15.690 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-24T18:15:15.804 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-24T18:15:15.811 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-24T18:15:15.811 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial020.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-24T18:17:39.770 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial020.front.sepia.ceph.com')] 2026-02-24T18:17:39.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T18:17:39.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T18:17:39.883 DEBUG:teuthology.orchestra.run.trial020:> true 2026-02-24T18:17:40.134 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T18:17:40.135 INFO:teuthology.run_tasks:Running task clock... 2026-02-24T18:17:40.147 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-24T18:17:40.147 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T18:17:40.148 DEBUG:teuthology.orchestra.run.trial020:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T18:17:40.197 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-24T18:17:40.197 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Command line: ntpd -gq 2026-02-24T18:17:40.197 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: ---------------------------------------------------- 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: ntp-4 is maintained by Network Time Foundation, 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: corporation. Support and training for ntp-4 are 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: available at https://www.nwtime.org/support 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: ---------------------------------------------------- 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: proto: precision = 0.030 usec (-25) 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: basedate set to 2022-02-04 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: gps base set to 2022-02-06 (week 2196) 2026-02-24T18:17:40.198 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-24T18:17:40.199 INFO:teuthology.orchestra.run.trial020.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-24T18:17:40.199 INFO:teuthology.orchestra.run.trial020.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-24T18:17:40.200 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: restrict ::: KOD does nothing without LIMITED. 2026-02-24T18:17:40.200 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Listen and drop on 0 v6wildcard [::]:123 2026-02-24T18:17:40.200 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-24T18:17:40.200 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Listen normally on 2 lo 127.0.0.1:123 2026-02-24T18:17:40.200 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Listen normally on 3 enp1s0f0 10.20.193.20:123 2026-02-24T18:17:40.200 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Listen normally on 4 lo [::1]:123 2026-02-24T18:17:40.200 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:62dc%2]:123 2026-02-24T18:17:40.201 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:40 ntpd[8815]: Listening on routing socket on fd #22 for interface updates 2026-02-24T18:17:43.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:43.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:43.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:43.198 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:43.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:43.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:43.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:43.218 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:43.237 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:43.238 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:43.238 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:43.238 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:43 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:44.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:44.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:44.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:44.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:44.206 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:44.206 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:44.206 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:44.207 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:44.240 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:44.241 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:44.241 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:44.241 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:44 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:45.196 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:45.196 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:45.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:45.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:45.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:45.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:45.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:45.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:45.237 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:45.238 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:45.238 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:45.238 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:45 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:46.196 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:46.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:46.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:46.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:46.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:46.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:46.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:46.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:46.244 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:46.244 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:46.245 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:46.245 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:46 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:47.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:47.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:47.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:47.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:47.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:47.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:47.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:47.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:47.237 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:47.237 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:47.237 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:47.237 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:47 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:48.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:48.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:48.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:48.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:48.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:48.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:48.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:48.204 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:48.244 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:48.244 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:48.244 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:48.244 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:48 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:49.196 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:49.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:49.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:49.197 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:49.216 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-24T18:17:49.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: can't open /var/log/ntpstats/rawstats.20260224: Permission denied 2026-02-24T18:17:49.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-24T18:17:49.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: can't open /var/log/ntpstats/peerstats.20260224: Permission denied 2026-02-24T18:17:49.217 INFO:teuthology.orchestra.run.trial020.stdout:24 Feb 18:17:49 ntpd[8815]: ntpd: time set -0.104747 s 2026-02-24T18:17:49.217 INFO:teuthology.orchestra.run.trial020.stdout:ntpd: time set -0.104747s 2026-02-24T18:17:49.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-24T18:17:49.217 INFO:teuthology.orchestra.run.trial020.stderr:24 Feb 18:17:49 ntpd[8815]: can't open /var/log/ntpstats/loopstats.20260224: Permission denied 2026-02-24T18:17:49.243 INFO:teuthology.orchestra.run.trial020.stdout:No association ID's returned 2026-02-24T18:17:49.244 INFO:teuthology.run_tasks:Running task install... 2026-02-24T18:17:49.261 DEBUG:teuthology.task.install:project ceph 2026-02-24T18:17:49.261 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-24T18:17:49.261 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-24T18:17:49.261 INFO:teuthology.task.install:Using flavor: default 2026-02-24T18:17:49.277 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T18:17:49.278 INFO:teuthology.task.install:extra packages: [] 2026-02-24T18:17:49.279 DEBUG:teuthology.orchestra.run.trial020:> sudo apt-key list | grep Ceph 2026-02-24T18:17:49.344 INFO:teuthology.orchestra.run.trial020.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-24T18:17:49.372 INFO:teuthology.orchestra.run.trial020.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-24T18:17:49.373 INFO:teuthology.orchestra.run.trial020.stdout:uid [ unknown] Ceph.com (release key) 2026-02-24T18:17:49.373 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-24T18:17:49.374 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-24T18:17:49.533 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-24T18:17:49.533 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-24T18:17:49.873 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:17:49.873 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-24T18:17:49.883 DEBUG:teuthology.orchestra.run.trial020:> sudo apt-get update 2026-02-24T18:17:50.207 INFO:teuthology.orchestra.run.trial020.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-24T18:17:50.259 INFO:teuthology.orchestra.run.trial020.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-24T18:17:50.312 INFO:teuthology.orchestra.run.trial020.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-24T18:17:50.361 INFO:teuthology.orchestra.run.trial020.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-24T18:17:50.402 INFO:teuthology.orchestra.run.trial020.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-24T18:17:50.488 INFO:teuthology.orchestra.run.trial020.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-24T18:17:50.580 INFO:teuthology.orchestra.run.trial020.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-24T18:17:50.672 INFO:teuthology.orchestra.run.trial020.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-24T18:17:50.814 INFO:teuthology.orchestra.run.trial020.stdout:Fetched 25.8 kB in 1s (33.5 kB/s) 2026-02-24T18:17:51.330 INFO:teuthology.orchestra.run.trial020.stdout:Reading package lists... 2026-02-24T18:17:51.342 DEBUG:teuthology.orchestra.run.trial020:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy 2026-02-24T18:17:51.375 INFO:teuthology.orchestra.run.trial020.stdout:Reading package lists... 2026-02-24T18:17:51.485 INFO:teuthology.orchestra.run.trial020.stdout:Building dependency tree... 2026-02-24T18:17:51.486 INFO:teuthology.orchestra.run.trial020.stdout:Reading state information... 2026-02-24T18:17:51.575 INFO:teuthology.orchestra.run.trial020.stdout:The following packages were automatically installed and are no longer required: 2026-02-24T18:17:51.575 INFO:teuthology.orchestra.run.trial020.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-24T18:17:51.575 INFO:teuthology.orchestra.run.trial020.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-24T18:17:51.575 INFO:teuthology.orchestra.run.trial020.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout:The following additional packages will be installed: 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-24T18:17:51.576 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout:Suggested packages: 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout:Recommended packages: 2026-02-24T18:17:51.577 INFO:teuthology.orchestra.run.trial020.stdout: btrfs-tools 2026-02-24T18:17:51.611 INFO:teuthology.orchestra.run.trial020.stdout:The following NEW packages will be installed: 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-24T18:17:51.612 INFO:teuthology.orchestra.run.trial020.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2026-02-24T18:17:51.613 INFO:teuthology.orchestra.run.trial020.stdout: xmlstarlet zip 2026-02-24T18:17:51.714 INFO:teuthology.orchestra.run.trial020.stdout:0 upgraded, 108 newly installed, 0 to remove and 14 not upgraded. 2026-02-24T18:17:51.714 INFO:teuthology.orchestra.run.trial020.stdout:Need to get 178 MB of archives. 2026-02-24T18:17:51.714 INFO:teuthology.orchestra.run.trial020.stdout:After this operation, 809 MB of additional disk space will be used. 2026-02-24T18:17:51.714 INFO:teuthology.orchestra.run.trial020.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-24T18:17:51.759 INFO:teuthology.orchestra.run.trial020.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-24T18:17:51.760 INFO:teuthology.orchestra.run.trial020.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-24T18:17:51.769 INFO:teuthology.orchestra.run.trial020.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-24T18:17:51.798 INFO:teuthology.orchestra.run.trial020.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-24T18:17:51.799 INFO:teuthology.orchestra.run.trial020.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-24T18:17:51.804 INFO:teuthology.orchestra.run.trial020.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-24T18:17:51.806 INFO:teuthology.orchestra.run.trial020.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-24T18:17:51.807 INFO:teuthology.orchestra.run.trial020.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-24T18:17:51.808 INFO:teuthology.orchestra.run.trial020.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-24T18:17:51.808 INFO:teuthology.orchestra.run.trial020.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-24T18:17:51.812 INFO:teuthology.orchestra.run.trial020.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-24T18:17:51.813 INFO:teuthology.orchestra.run.trial020.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-24T18:17:51.815 INFO:teuthology.orchestra.run.trial020.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-24T18:17:51.818 INFO:teuthology.orchestra.run.trial020.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-24T18:17:51.819 INFO:teuthology.orchestra.run.trial020.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-24T18:17:51.820 INFO:teuthology.orchestra.run.trial020.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-24T18:17:51.821 INFO:teuthology.orchestra.run.trial020.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-24T18:17:51.822 INFO:teuthology.orchestra.run.trial020.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-24T18:17:51.828 INFO:teuthology.orchestra.run.trial020.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-24T18:17:51.828 INFO:teuthology.orchestra.run.trial020.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-24T18:17:51.829 INFO:teuthology.orchestra.run.trial020.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-24T18:17:51.829 INFO:teuthology.orchestra.run.trial020.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-24T18:17:51.829 INFO:teuthology.orchestra.run.trial020.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-24T18:17:51.837 INFO:teuthology.orchestra.run.trial020.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-24T18:17:51.838 INFO:teuthology.orchestra.run.trial020.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-24T18:17:51.838 INFO:teuthology.orchestra.run.trial020.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-24T18:17:51.838 INFO:teuthology.orchestra.run.trial020.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-24T18:17:51.840 INFO:teuthology.orchestra.run.trial020.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-24T18:17:51.847 INFO:teuthology.orchestra.run.trial020.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-24T18:17:51.847 INFO:teuthology.orchestra.run.trial020.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-24T18:17:51.848 INFO:teuthology.orchestra.run.trial020.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-24T18:17:51.848 INFO:teuthology.orchestra.run.trial020.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-24T18:17:51.848 INFO:teuthology.orchestra.run.trial020.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-24T18:17:51.857 INFO:teuthology.orchestra.run.trial020.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-24T18:17:51.858 INFO:teuthology.orchestra.run.trial020.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-24T18:17:51.858 INFO:teuthology.orchestra.run.trial020.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-24T18:17:51.861 INFO:teuthology.orchestra.run.trial020.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-24T18:17:51.861 INFO:teuthology.orchestra.run.trial020.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-24T18:17:51.866 INFO:teuthology.orchestra.run.trial020.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-24T18:17:51.867 INFO:teuthology.orchestra.run.trial020.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-24T18:17:51.867 INFO:teuthology.orchestra.run.trial020.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-24T18:17:51.869 INFO:teuthology.orchestra.run.trial020.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-24T18:17:51.870 INFO:teuthology.orchestra.run.trial020.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-24T18:17:51.876 INFO:teuthology.orchestra.run.trial020.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-24T18:17:51.876 INFO:teuthology.orchestra.run.trial020.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-24T18:17:51.877 INFO:teuthology.orchestra.run.trial020.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-24T18:17:51.891 INFO:teuthology.orchestra.run.trial020.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-24T18:17:51.892 INFO:teuthology.orchestra.run.trial020.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-24T18:17:51.893 INFO:teuthology.orchestra.run.trial020.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-24T18:17:51.899 INFO:teuthology.orchestra.run.trial020.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-24T18:17:51.903 INFO:teuthology.orchestra.run.trial020.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-24T18:17:51.903 INFO:teuthology.orchestra.run.trial020.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-24T18:17:51.903 INFO:teuthology.orchestra.run.trial020.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-24T18:17:51.904 INFO:teuthology.orchestra.run.trial020.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-24T18:17:51.904 INFO:teuthology.orchestra.run.trial020.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-24T18:17:51.904 INFO:teuthology.orchestra.run.trial020.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-24T18:17:51.905 INFO:teuthology.orchestra.run.trial020.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-24T18:17:51.906 INFO:teuthology.orchestra.run.trial020.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-24T18:17:51.906 INFO:teuthology.orchestra.run.trial020.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-24T18:17:51.914 INFO:teuthology.orchestra.run.trial020.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-24T18:17:51.915 INFO:teuthology.orchestra.run.trial020.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-24T18:17:51.917 INFO:teuthology.orchestra.run.trial020.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-24T18:17:51.917 INFO:teuthology.orchestra.run.trial020.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-24T18:17:51.917 INFO:teuthology.orchestra.run.trial020.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-24T18:17:51.923 INFO:teuthology.orchestra.run.trial020.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-24T18:17:51.924 INFO:teuthology.orchestra.run.trial020.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-24T18:17:51.927 INFO:teuthology.orchestra.run.trial020.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-24T18:17:51.927 INFO:teuthology.orchestra.run.trial020.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-24T18:17:51.927 INFO:teuthology.orchestra.run.trial020.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-24T18:17:51.933 INFO:teuthology.orchestra.run.trial020.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-24T18:17:51.934 INFO:teuthology.orchestra.run.trial020.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-24T18:17:51.934 INFO:teuthology.orchestra.run.trial020.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-24T18:17:51.938 INFO:teuthology.orchestra.run.trial020.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-24T18:17:51.938 INFO:teuthology.orchestra.run.trial020.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-24T18:17:51.943 INFO:teuthology.orchestra.run.trial020.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-24T18:17:51.944 INFO:teuthology.orchestra.run.trial020.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-24T18:17:51.944 INFO:teuthology.orchestra.run.trial020.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-24T18:17:52.639 INFO:teuthology.orchestra.run.trial020.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-24T18:17:52.836 INFO:teuthology.orchestra.run.trial020.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-24T18:17:52.909 INFO:teuthology.orchestra.run.trial020.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-24T18:17:52.945 INFO:teuthology.orchestra.run.trial020.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-24T18:17:52.945 INFO:teuthology.orchestra.run.trial020.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-24T18:17:52.949 INFO:teuthology.orchestra.run.trial020.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-24T18:17:52.967 INFO:teuthology.orchestra.run.trial020.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-24T18:17:52.977 INFO:teuthology.orchestra.run.trial020.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-24T18:17:53.270 INFO:teuthology.orchestra.run.trial020.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-24T18:17:53.283 INFO:teuthology.orchestra.run.trial020.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-24T18:17:53.296 INFO:teuthology.orchestra.run.trial020.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-24T18:17:54.367 INFO:teuthology.orchestra.run.trial020.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-24T18:17:54.680 INFO:teuthology.orchestra.run.trial020.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-24T18:17:54.760 INFO:teuthology.orchestra.run.trial020.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-24T18:17:54.761 INFO:teuthology.orchestra.run.trial020.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-24T18:17:54.793 INFO:teuthology.orchestra.run.trial020.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-24T18:17:55.144 INFO:teuthology.orchestra.run.trial020.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-24T18:17:56.549 INFO:teuthology.orchestra.run.trial020.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-24T18:17:56.549 INFO:teuthology.orchestra.run.trial020.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-24T18:17:56.616 INFO:teuthology.orchestra.run.trial020.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-24T18:17:56.768 INFO:teuthology.orchestra.run.trial020.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-24T18:17:56.831 INFO:teuthology.orchestra.run.trial020.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-24T18:17:56.839 INFO:teuthology.orchestra.run.trial020.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-24T18:17:56.986 INFO:teuthology.orchestra.run.trial020.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-24T18:17:57.622 INFO:teuthology.orchestra.run.trial020.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-24T18:17:57.622 INFO:teuthology.orchestra.run.trial020.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-24T18:18:02.637 INFO:teuthology.orchestra.run.trial020.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-24T18:18:02.637 INFO:teuthology.orchestra.run.trial020.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-24T18:18:02.638 INFO:teuthology.orchestra.run.trial020.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-24T18:18:04.341 INFO:teuthology.orchestra.run.trial020.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-24T18:18:04.663 INFO:teuthology.orchestra.run.trial020.stdout:Fetched 178 MB in 13s (14.0 MB/s) 2026-02-24T18:18:04.679 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-24T18:18:04.844 INFO:teuthology.orchestra.run.trial020.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 202926 files and directories currently installed.) 2026-02-24T18:18:04.847 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-24T18:18:04.848 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-24T18:18:04.870 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-24T18:18:04.885 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-24T18:18:04.885 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-24T18:18:04.901 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-24T18:18:04.915 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-24T18:18:04.915 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-24T18:18:04.934 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-24T18:18:04.944 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-24T18:18:04.949 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:18:04.980 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-24T18:18:04.990 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-24T18:18:04.990 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:18:05.007 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-24T18:18:05.017 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-24T18:18:05.018 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:18:05.038 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-24T18:18:05.048 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-24T18:18:05.048 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-24T18:18:05.066 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package librados2. 2026-02-24T18:18:05.077 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.077 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.118 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libnbd0. 2026-02-24T18:18:05.128 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-24T18:18:05.129 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-24T18:18:05.144 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package librbd1. 2026-02-24T18:18:05.153 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.154 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.194 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libcephfs2. 2026-02-24T18:18:05.204 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.205 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.226 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-rados. 2026-02-24T18:18:05.237 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.237 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.255 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-24T18:18:05.265 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:05.265 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.280 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-cephfs. 2026-02-24T18:18:05.290 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.290 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.306 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-24T18:18:05.317 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:05.317 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.335 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-24T18:18:05.345 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-24T18:18:05.345 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-24T18:18:05.361 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-prettytable. 2026-02-24T18:18:05.371 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-24T18:18:05.372 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-24T18:18:05.387 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-rbd. 2026-02-24T18:18:05.397 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.397 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.415 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-24T18:18:05.425 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-24T18:18:05.425 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-24T18:18:05.445 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-24T18:18:05.454 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-24T18:18:05.455 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-24T18:18:05.472 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-24T18:18:05.482 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-24T18:18:05.483 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-24T18:18:05.500 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package lua5.1. 2026-02-24T18:18:05.510 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-24T18:18:05.510 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-24T18:18:05.530 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package lua-any. 2026-02-24T18:18:05.539 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-24T18:18:05.540 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-24T18:18:05.555 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package zip. 2026-02-24T18:18:05.565 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-24T18:18:05.565 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking zip (3.0-12build2) ... 2026-02-24T18:18:05.582 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package unzip. 2026-02-24T18:18:05.592 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-24T18:18:05.592 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-24T18:18:05.610 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package luarocks. 2026-02-24T18:18:05.620 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-24T18:18:05.620 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-24T18:18:05.659 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package librgw2. 2026-02-24T18:18:05.669 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.669 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.741 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-rgw. 2026-02-24T18:18:05.752 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.752 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.768 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-24T18:18:05.777 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-24T18:18:05.778 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-24T18:18:05.793 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libradosstriper1. 2026-02-24T18:18:05.804 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.804 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:05.824 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-common. 2026-02-24T18:18:05.833 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:05.834 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:06.041 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-base. 2026-02-24T18:18:06.051 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:06.056 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:06.120 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-24T18:18:06.129 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-24T18:18:06.130 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-24T18:18:06.144 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-cheroot. 2026-02-24T18:18:06.155 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-24T18:18:06.156 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-24T18:18:06.174 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-24T18:18:06.184 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-24T18:18:06.184 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-24T18:18:06.200 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-24T18:18:06.210 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-24T18:18:06.210 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-24T18:18:06.226 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-24T18:18:06.236 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-24T18:18:06.236 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-24T18:18:06.251 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-tempora. 2026-02-24T18:18:06.262 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-24T18:18:06.262 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-24T18:18:06.278 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-portend. 2026-02-24T18:18:06.288 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-24T18:18:06.289 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-24T18:18:06.304 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-24T18:18:06.314 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-24T18:18:06.314 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-24T18:18:06.329 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-24T18:18:06.339 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-24T18:18:06.340 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-24T18:18:06.363 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-natsort. 2026-02-24T18:18:06.374 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-24T18:18:06.374 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-24T18:18:06.390 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-logutils. 2026-02-24T18:18:06.400 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-24T18:18:06.400 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-24T18:18:06.416 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-mako. 2026-02-24T18:18:06.426 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-24T18:18:06.426 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-24T18:18:06.445 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-24T18:18:06.455 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-24T18:18:06.455 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-24T18:18:06.471 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-24T18:18:06.481 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-24T18:18:06.482 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-24T18:18:06.496 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-webob. 2026-02-24T18:18:06.506 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-24T18:18:06.506 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-24T18:18:06.524 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-waitress. 2026-02-24T18:18:06.534 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-24T18:18:06.536 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-24T18:18:06.552 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-tempita. 2026-02-24T18:18:06.562 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-24T18:18:06.562 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-24T18:18:06.578 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-paste. 2026-02-24T18:18:06.588 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-24T18:18:06.588 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-24T18:18:06.616 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-24T18:18:06.626 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-24T18:18:06.626 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-24T18:18:06.640 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-24T18:18:06.650 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-24T18:18:06.651 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-24T18:18:06.667 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-webtest. 2026-02-24T18:18:06.677 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-24T18:18:06.677 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-24T18:18:06.694 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-pecan. 2026-02-24T18:18:06.704 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-24T18:18:06.704 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-24T18:18:06.730 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-24T18:18:06.739 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-24T18:18:06.740 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-24T18:18:06.762 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-24T18:18:06.772 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:06.772 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:06.801 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-24T18:18:06.810 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:06.811 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:06.827 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mgr. 2026-02-24T18:18:06.837 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:06.837 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:06.860 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mon. 2026-02-24T18:18:06.869 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:06.870 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:06.929 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-24T18:18:06.939 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-24T18:18:06.939 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-24T18:18:06.956 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-osd. 2026-02-24T18:18:06.966 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:06.966 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.121 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph. 2026-02-24T18:18:07.132 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:07.133 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.147 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-fuse. 2026-02-24T18:18:07.157 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:07.157 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.182 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mds. 2026-02-24T18:18:07.192 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:07.193 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.226 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package cephadm. 2026-02-24T18:18:07.236 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:07.236 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.253 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-24T18:18:07.263 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-24T18:18:07.263 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-24T18:18:07.286 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-24T18:18:07.295 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:07.296 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.316 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-24T18:18:07.326 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-24T18:18:07.327 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-24T18:18:07.342 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-routes. 2026-02-24T18:18:07.352 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-24T18:18:07.352 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-24T18:18:07.373 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-24T18:18:07.383 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:07.383 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.586 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-24T18:18:07.596 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-24T18:18:07.597 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-24T18:18:07.636 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-joblib. 2026-02-24T18:18:07.646 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-24T18:18:07.647 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-24T18:18:07.674 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-24T18:18:07.684 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-24T18:18:07.685 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-24T18:18:07.700 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-sklearn. 2026-02-24T18:18:07.710 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-24T18:18:07.710 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-24T18:18:07.796 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-24T18:18:07.805 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:07.806 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:07.977 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-cachetools. 2026-02-24T18:18:07.987 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-24T18:18:07.988 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-24T18:18:08.004 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-rsa. 2026-02-24T18:18:08.014 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-24T18:18:08.014 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-24T18:18:08.032 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-google-auth. 2026-02-24T18:18:08.042 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-24T18:18:08.043 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-24T18:18:08.061 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-24T18:18:08.071 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-24T18:18:08.071 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-24T18:18:08.086 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-websocket. 2026-02-24T18:18:08.097 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-24T18:18:08.098 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-24T18:18:08.116 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-24T18:18:08.126 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-24T18:18:08.126 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-24T18:18:08.224 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-24T18:18:08.235 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:08.235 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:08.250 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-24T18:18:08.260 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-24T18:18:08.268 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-24T18:18:08.284 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-24T18:18:08.294 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-24T18:18:08.294 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-24T18:18:08.309 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package jq. 2026-02-24T18:18:08.320 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-24T18:18:08.320 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-24T18:18:08.335 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package socat. 2026-02-24T18:18:08.345 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-24T18:18:08.346 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-24T18:18:08.365 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package xmlstarlet. 2026-02-24T18:18:08.376 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-24T18:18:08.376 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-24T18:18:08.413 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-test. 2026-02-24T18:18:08.423 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:08.424 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:08.909 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package ceph-volume. 2026-02-24T18:18:08.920 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-24T18:18:08.921 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:08.944 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-24T18:18:08.954 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:08.954 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:08.971 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-24T18:18:08.982 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-24T18:18:08.982 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-24T18:18:09.004 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-24T18:18:09.014 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-24T18:18:09.014 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-24T18:18:09.032 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package nvme-cli. 2026-02-24T18:18:09.043 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-24T18:18:09.043 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-24T18:18:09.077 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package pkg-config. 2026-02-24T18:18:09.087 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-24T18:18:09.087 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-24T18:18:09.104 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-24T18:18:09.115 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-24T18:18:09.115 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-24T18:18:09.145 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-24T18:18:09.155 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-24T18:18:09.155 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-24T18:18:09.171 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-pastescript. 2026-02-24T18:18:09.182 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-24T18:18:09.182 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-24T18:18:09.202 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-pluggy. 2026-02-24T18:18:09.213 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-24T18:18:09.213 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-24T18:18:09.229 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-psutil. 2026-02-24T18:18:09.240 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-24T18:18:09.240 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-24T18:18:09.259 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-py. 2026-02-24T18:18:09.269 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-24T18:18:09.269 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-24T18:18:09.290 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-pygments. 2026-02-24T18:18:09.301 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-24T18:18:09.301 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-24T18:18:09.343 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-24T18:18:09.353 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-24T18:18:09.354 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-24T18:18:09.370 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-toml. 2026-02-24T18:18:09.380 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-24T18:18:09.381 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-24T18:18:09.396 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-pytest. 2026-02-24T18:18:09.406 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-24T18:18:09.407 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-24T18:18:09.428 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package python3-simplejson. 2026-02-24T18:18:09.439 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-24T18:18:09.439 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-24T18:18:09.458 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-24T18:18:09.468 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-24T18:18:09.469 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-24T18:18:09.537 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package radosgw. 2026-02-24T18:18:09.548 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:09.548 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:09.664 INFO:teuthology.orchestra.run.trial020.stdout:Selecting previously unselected package rbd-fuse. 2026-02-24T18:18:09.674 INFO:teuthology.orchestra.run.trial020.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-24T18:18:09.675 INFO:teuthology.orchestra.run.trial020.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:09.705 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-24T18:18:09.772 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-24T18:18:09.773 INFO:teuthology.orchestra.run.trial020.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-24T18:18:09.846 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-24T18:18:10.118 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-24T18:18:10.467 INFO:teuthology.orchestra.run.trial020.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-24T18:18:10.478 INFO:teuthology.orchestra.run.trial020.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:10.521 INFO:teuthology.orchestra.run.trial020.stdout:Adding system user cephadm....done 2026-02-24T18:18:10.532 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-24T18:18:10.602 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-24T18:18:10.676 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-24T18:18:10.677 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-24T18:18:10.748 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-24T18:18:10.823 INFO:teuthology.orchestra.run.trial020.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-24T18:18:10.824 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-24T18:18:10.918 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-24T18:18:11.042 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-24T18:18:11.118 INFO:teuthology.orchestra.run.trial020.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-24T18:18:11.130 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-24T18:18:11.205 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-24T18:18:11.278 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:11.347 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-24T18:18:11.348 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-24T18:18:11.349 INFO:teuthology.orchestra.run.trial020.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-24T18:18:11.350 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-24T18:18:11.351 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-24T18:18:11.351 INFO:teuthology.orchestra.run.trial020.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-24T18:18:11.368 INFO:teuthology.orchestra.run.trial020.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-24T18:18:11.370 INFO:teuthology.orchestra.run.trial020.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-24T18:18:11.371 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-24T18:18:11.372 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-24T18:18:11.493 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-24T18:18:11.569 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-24T18:18:11.650 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-24T18:18:11.736 INFO:teuthology.orchestra.run.trial020.stdout:Setting up zip (3.0-12build2) ... 2026-02-24T18:18:11.737 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-24T18:18:11.982 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-24T18:18:12.059 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-24T18:18:12.060 INFO:teuthology.orchestra.run.trial020.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-24T18:18:12.061 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-24T18:18:12.154 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-24T18:18:12.277 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-24T18:18:12.407 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-24T18:18:12.488 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-24T18:18:12.592 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-24T18:18:12.667 INFO:teuthology.orchestra.run.trial020.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-24T18:18:12.667 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:12.750 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-24T18:18:13.241 INFO:teuthology.orchestra.run.trial020.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-24T18:18:13.263 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:18:13.268 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-24T18:18:13.346 INFO:teuthology.orchestra.run.trial020.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-24T18:18:13.347 INFO:teuthology.orchestra.run.trial020.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-24T18:18:13.348 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-24T18:18:13.423 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-24T18:18:13.496 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:18:13.497 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-24T18:18:13.573 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-24T18:18:13.644 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-24T18:18:13.719 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-24T18:18:13.798 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-24T18:18:13.870 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-24T18:18:13.952 INFO:teuthology.orchestra.run.trial020.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-24T18:18:13.953 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-24T18:18:14.038 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-24T18:18:14.039 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-24T18:18:14.114 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-24T18:18:14.194 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-24T18:18:14.278 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-24T18:18:14.349 INFO:teuthology.orchestra.run.trial020.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-24T18:18:14.350 INFO:teuthology.orchestra.run.trial020.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-24T18:18:14.351 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-24T18:18:14.352 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-24T18:18:14.485 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-24T18:18:14.561 INFO:teuthology.orchestra.run.trial020.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-24T18:18:14.562 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-24T18:18:14.636 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-24T18:18:14.636 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-24T18:18:14.717 INFO:teuthology.orchestra.run.trial020.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-24T18:18:14.718 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-24T18:18:14.798 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-24T18:18:14.937 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-24T18:18:15.026 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-24T18:18:15.144 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-24T18:18:15.145 INFO:teuthology.orchestra.run.trial020.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:15.146 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:15.147 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-24T18:18:15.662 INFO:teuthology.orchestra.run.trial020.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-24T18:18:15.673 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:15.674 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:15.675 INFO:teuthology.orchestra.run.trial020.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:15.676 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:15.677 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:15.745 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-24T18:18:15.746 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-24T18:18:16.117 INFO:teuthology.orchestra.run.trial020.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.118 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.119 INFO:teuthology.orchestra.run.trial020.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.120 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.120 INFO:teuthology.orchestra.run.trial020.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.121 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.122 INFO:teuthology.orchestra.run.trial020.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.123 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.152 INFO:teuthology.orchestra.run.trial020.stdout:Adding group ceph....done 2026-02-24T18:18:16.187 INFO:teuthology.orchestra.run.trial020.stdout:Adding system user ceph....done 2026-02-24T18:18:16.196 INFO:teuthology.orchestra.run.trial020.stdout:Setting system user ceph properties....done 2026-02-24T18:18:16.203 INFO:teuthology.orchestra.run.trial020.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-24T18:18:16.290 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-24T18:18:16.566 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-24T18:18:16.926 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:16.927 INFO:teuthology.orchestra.run.trial020.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:17.175 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-24T18:18:17.176 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-24T18:18:17.513 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:17.599 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-24T18:18:17.931 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:17.991 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-24T18:18:17.991 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-24T18:18:18.333 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:18.411 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-24T18:18:18.411 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-24T18:18:18.728 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:18.822 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-24T18:18:18.822 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-24T18:18:19.189 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:19.190 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:19.200 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:19.272 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-24T18:18:19.272 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-24T18:18:19.619 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:19.631 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:19.631 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:19.641 INFO:teuthology.orchestra.run.trial020.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-24T18:18:19.751 INFO:teuthology.orchestra.run.trial020.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-24T18:18:19.759 INFO:teuthology.orchestra.run.trial020.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-24T18:18:20.018 INFO:teuthology.orchestra.run.trial020.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-24T18:18:20.105 INFO:teuthology.orchestra.run.trial020.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-24T18:18:20.615 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:20.615 INFO:teuthology.orchestra.run.trial020.stdout:Running kernel seems to be up-to-date. 2026-02-24T18:18:20.615 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout:The processor microcode seems to be up-to-date. 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout:No services need to be restarted. 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout:No containers need to be restarted. 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout:No user sessions are running outdated binaries. 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:20.616 INFO:teuthology.orchestra.run.trial020.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-24T18:18:21.447 INFO:teuthology.orchestra.run.trial020.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-24T18:18:21.454 DEBUG:teuthology.parallel:result is None 2026-02-24T18:18:21.454 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-24T18:18:21.590 DEBUG:teuthology.orchestra.run.trial020:> dpkg-query -W -f '${Version}' ceph 2026-02-24T18:18:21.599 INFO:teuthology.orchestra.run.trial020.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-24T18:18:21.600 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-24T18:18:21.600 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-24T18:18:21.602 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-24T18:18:21.602 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:21.602 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T18:18:21.651 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-24T18:18:21.651 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:21.651 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T18:18:21.702 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T18:18:21.750 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-24T18:18:21.751 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:21.751 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T18:18:21.798 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T18:18:21.846 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-24T18:18:21.846 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:21.846 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T18:18:21.894 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T18:18:21.943 INFO:teuthology.run_tasks:Running task ceph... 2026-02-24T18:18:22.030 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-24T18:18:22.030 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 777 /var/log/ceph 2026-02-24T18:18:22.039 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-24T18:18:22.039 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-24T18:18:22.089 INFO:tasks.ceph:Creating extra log directories... 2026-02-24T18:18:22.089 DEBUG:teuthology.orchestra.run.trial020:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-24T18:18:22.147 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-24T18:18:22.147 INFO:tasks.ceph:config {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-24T18:18:22.147 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}', 'email': None, 'first_in_suite': False, 'flavor': 'default', 'job_id': '68642', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5907, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial020.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFP9n/0BnzkUCBW+t12y0FKb2phILCpQ7wUwUFW3B67xkJt8UuwtRzvs6e7IaLPAk1pCfqK63RccEY0WiPzfZbg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['cephtool/test_kvstore_tool.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'reimage-ssh-down', 'teuthology_sha1': 'e6d9099ce03dc21193fda653159ec82465c3a94a', 'timestamp': '2026-02-24_16:22:43', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-24T18:18:22.148 DEBUG:teuthology.orchestra.run.trial020:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-24T18:18:22.186 DEBUG:teuthology.orchestra.run.trial020:> sudo install -d -m0777 -- /var/run/ceph 2026-02-24T18:18:22.235 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:22.235 DEBUG:teuthology.orchestra.run.trial020:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T18:18:22.279 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T18:18:22.279 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_1 2026-02-24T18:18:22.326 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T18:18:22.326 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:18:22.326 INFO:teuthology.orchestra.run.trial020.stdout:Device: 5h/5d Inode: 906 Links: 1 2026-02-24T18:18:22.327 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:18:22.327 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 18:17:27.856009587 +0000 2026-02-24T18:18:22.327 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 18:17:27.692012376 +0000 2026-02-24T18:18:22.327 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 18:17:27.692012376 +0000 2026-02-24T18:18:22.327 INFO:teuthology.orchestra.run.trial020.stdout: Birth: - 2026-02-24T18:18:22.327 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T18:18:22.376 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T18:18:22.376 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T18:18:22.376 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.000193088 s, 2.7 MB/s 2026-02-24T18:18:22.377 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T18:18:22.424 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_2 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout:Device: 5h/5d Inode: 929 Links: 1 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 18:17:28.164004348 +0000 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 18:17:28.008007003 +0000 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 18:17:28.008007003 +0000 2026-02-24T18:18:22.470 INFO:teuthology.orchestra.run.trial020.stdout: Birth: - 2026-02-24T18:18:22.470 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T18:18:22.518 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T18:18:22.518 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T18:18:22.518 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.00017235 s, 3.0 MB/s 2026-02-24T18:18:22.519 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T18:18:22.569 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_3 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout:Device: 5h/5d Inode: 954 Links: 1 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 18:17:28.479998978 +0000 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 18:17:28.312001832 +0000 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 18:17:28.312001832 +0000 2026-02-24T18:18:22.620 INFO:teuthology.orchestra.run.trial020.stdout: Birth: - 2026-02-24T18:18:22.620 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T18:18:22.669 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T18:18:22.670 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T18:18:22.670 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.000174123 s, 2.9 MB/s 2026-02-24T18:18:22.670 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T18:18:22.715 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_4 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout:Device: 5h/5d Inode: 979 Links: 1 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 18:17:28.799993534 +0000 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 18:17:28.627996462 +0000 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 18:17:28.627996462 +0000 2026-02-24T18:18:22.762 INFO:teuthology.orchestra.run.trial020.stdout: Birth: - 2026-02-24T18:18:22.763 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T18:18:22.810 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T18:18:22.810 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T18:18:22.810 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.000153395 s, 3.3 MB/s 2026-02-24T18:18:22.811 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T18:18:22.859 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T18:18:22.859 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial020.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T18:18:22.859 INFO:tasks.ceph:Generating config... 2026-02-24T18:18:22.860 INFO:tasks.ceph:[global] mon election default strategy = 3 2026-02-24T18:18:22.861 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-24T18:18:22.861 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-24T18:18:22.861 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-24T18:18:22.861 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-24T18:18:22.861 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-24T18:18:22.861 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-24T18:18:22.861 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-24T18:18:22.861 INFO:tasks.ceph:Setting up mon.a... 2026-02-24T18:18:22.861 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-24T18:18:22.918 INFO:teuthology.orchestra.run.trial020.stdout:creating /etc/ceph/ceph.keyring 2026-02-24T18:18:22.922 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-24T18:18:22.982 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T18:18:23.030 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.20')] 2026-02-24T18:18:23.031 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.20', 'mon election default strategy': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-24T18:18:23.031 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:23.031 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T18:18:23.075 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.20 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:18:23.134 INFO:teuthology.orchestra.run.trial020.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:monmaptool: generated fsid 135e2b35-ac06-4607-a4ba-6aa45d264e7b 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:setting min_mon_release = quincy 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:epoch 0 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:fsid 135e2b35-ac06-4607-a4ba-6aa45d264e7b 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:last_changed 2026-02-24T18:18:23.142163+0000 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:created 2026-02-24T18:18:23.142163+0000 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:min_mon_release 17 (quincy) 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:election_strategy: 3 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:0: [v2:10.20.193.20:3300/0,v1:10.20.193.20:6789/0] mon.a 2026-02-24T18:18:23.135 INFO:teuthology.orchestra.run.trial020.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-24T18:18:23.136 DEBUG:teuthology.orchestra.run.trial020:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T18:18:23.182 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 135e2b35-ac06-4607-a4ba-6aa45d264e7b... 2026-02-24T18:18:23.184 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout:[global] 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: chdir = "" 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: auth supported = cephx 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: filestore xattr use omap = true 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: mon clock drift allowed = 1.000 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: osd crush chooseleaf type = 0 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: auth debug = true 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: ms die on old message = true 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: ms die on bug = true 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: mon pg warn max object skew = 0 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: # disable pg_autoscaler by default for new pools 2026-02-24T18:18:23.242 INFO:teuthology.orchestra.run.trial020.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: osd pool default size = 2 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon osd allow primary affinity = true 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon osd allow pg remap = true 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on legacy crush tunables = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on crush straw calc version zero = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on no sortbitwise = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on osd down out interval zero = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on too few osds = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon_allow_pool_size_one = true 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: osd default data pool replay window = 5 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon allow pool delete = true 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon cluster log file level = debug 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: debug asserts on shutdown = true 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon health detail to clog = false 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon host = 10.20.193.20 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: mon election default strategy = 3 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: fsid = 135e2b35-ac06-4607-a4ba-6aa45d264e7b 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout:[osd] 2026-02-24T18:18:23.243 INFO:teuthology.orchestra.run.trial020.stdout: osd journal size = 100 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd scrub load threshold = 5.0 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd scrub max interval = 600 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd mclock profile = high_recovery_ops 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd mclock skip benchmark = True 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd recover clone overlap = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd recovery max chunk = 1048576 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd debug shutdown = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd debug op order = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd debug verify stray on activate = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd debug trim objects = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd open classes on start = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd debug pg log writeout = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd deep scrub update digest min age = 30 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd map max advance = 10 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: journal zero on create = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: filestore ondisk finisher threads = 3 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: filestore apply finisher threads = 3 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: bdev debug aio = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: osd debug misdirected ops = true 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: bluestore zero block detection = True 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: debug ms = 1 2026-02-24T18:18:23.244 INFO:teuthology.orchestra.run.trial020.stdout: debug osd = 20 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: osd debug verify cached snaps = True 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: osd debug verify missing on start = True 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: osd mclock override recovery settings = True 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: osd op queue = debug_random 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: osd op queue cut off = debug_random 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout:[mgr] 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug ms = 1 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug mgr = 20 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug mon = 20 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug auth = 20 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min pgs per osd = 4 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min bytes per osd = 10 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mgr/telemetry/nag = false 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout:[mon] 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug ms = 1 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug mon = 20 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug paxos = 20 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: debug auth = 20 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon data avail warn = 5 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon mgr mkfs grace = 240 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min pgs per osd = 4 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon osd reporter subtree level = osd 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon osd prime pg temp = true 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min bytes per osd = 10 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: auth mon ticket ttl = 660 # 11m 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: auth service ticket ttl = 240 # 4m 2026-02-24T18:18:23.245 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: # don't complain about insecure global_id in the test suite 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: # 1m isn't quite enough 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: mon_down_mkfs_grace = 2m 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_filestore_osds = false 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: mon scrub interval = 300 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout:[client] 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: rgw cache enabled = true 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: rgw enable ops log = true 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: rgw enable usage log = true 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-24T18:18:23.246 INFO:teuthology.orchestra.run.trial020.stdout:[mon.a] 2026-02-24T18:18:23.250 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-24T18:18:23.250 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-24T18:18:23.312 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-24T18:18:23.312 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:23.312 DEBUG:teuthology.orchestra.run.trial020:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-24T18:18:23.358 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:23.358 DEBUG:teuthology.orchestra.run.trial020:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-24T18:18:23.403 INFO:tasks.ceph:Sending monmap to node ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:18:23.403 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:23.403 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-24T18:18:23.403 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T18:18:23.460 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:23.460 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:18:23.506 INFO:tasks.ceph:Setting up mon nodes... 2026-02-24T18:18:23.506 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-24T18:18:23.506 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T18:18:23.572 INFO:teuthology.orchestra.run.trial020.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T18:18:23.574 INFO:tasks.ceph:Setting up mds nodes... 2026-02-24T18:18:23.574 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-24T18:18:23.574 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-24T18:18:23.631 INFO:teuthology.orchestra.run.trial020.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-24T18:18:23.638 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-24T18:18:23.638 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial020.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T18:18:23.638 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-24T18:18:23.688 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T18:18:23.688 INFO:tasks.ceph:role: osd.0 2026-02-24T18:18:23.688 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:18:23.688 DEBUG:teuthology.orchestra.run.trial020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-24T18:18:23.740 INFO:teuthology.orchestra.run.trial020.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T18:18:23.740 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T18:18:23.740 INFO:teuthology.orchestra.run.trial020.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T18:18:23.741 INFO:teuthology.orchestra.run.trial020.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-24T18:18:23.741 INFO:teuthology.orchestra.run.trial020.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T18:18:23.741 INFO:teuthology.orchestra.run.trial020.stdout: = sunit=0 swidth=0 blks 2026-02-24T18:18:23.741 INFO:teuthology.orchestra.run.trial020.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T18:18:23.741 INFO:teuthology.orchestra.run.trial020.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T18:18:23.741 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T18:18:23.741 INFO:teuthology.orchestra.run.trial020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T18:18:24.312 INFO:teuthology.orchestra.run.trial020.stdout:Discarding blocks...Done. 2026-02-24T18:18:24.314 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial020.front.sepia.ceph.com -o noatime 2026-02-24T18:18:24.314 DEBUG:teuthology.orchestra.run.trial020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-24T18:18:24.368 DEBUG:teuthology.orchestra.run.trial020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-24T18:18:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:18:24.417 INFO:teuthology.orchestra.run.trial020.stderr:sudo: /sbin/restorecon: command not found 2026-02-24T18:18:24.418 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-24T18:18:24.467 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T18:18:24.467 INFO:tasks.ceph:role: osd.1 2026-02-24T18:18:24.467 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:18:24.467 DEBUG:teuthology.orchestra.run.trial020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-24T18:18:24.520 INFO:teuthology.orchestra.run.trial020.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T18:18:24.520 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout: = sunit=0 swidth=0 blks 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T18:18:24.521 INFO:teuthology.orchestra.run.trial020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T18:18:25.086 INFO:teuthology.orchestra.run.trial020.stdout:Discarding blocks...Done. 2026-02-24T18:18:25.088 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial020.front.sepia.ceph.com -o noatime 2026-02-24T18:18:25.088 DEBUG:teuthology.orchestra.run.trial020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-24T18:18:25.145 DEBUG:teuthology.orchestra.run.trial020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-24T18:18:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:18:25.194 INFO:teuthology.orchestra.run.trial020.stderr:sudo: /sbin/restorecon: command not found 2026-02-24T18:18:25.195 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-24T18:18:25.243 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T18:18:25.243 INFO:tasks.ceph:role: osd.2 2026-02-24T18:18:25.243 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:18:25.243 DEBUG:teuthology.orchestra.run.trial020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout: = sunit=0 swidth=0 blks 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T18:18:25.292 INFO:teuthology.orchestra.run.trial020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T18:18:25.856 INFO:teuthology.orchestra.run.trial020.stdout:Discarding blocks...Done. 2026-02-24T18:18:25.858 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial020.front.sepia.ceph.com -o noatime 2026-02-24T18:18:25.858 DEBUG:teuthology.orchestra.run.trial020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-24T18:18:25.911 DEBUG:teuthology.orchestra.run.trial020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-24T18:18:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:18:25.961 INFO:teuthology.orchestra.run.trial020.stderr:sudo: /sbin/restorecon: command not found 2026-02-24T18:18:25.962 DEBUG:teuthology.orchestra.run.trial020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:18:26.024 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:26.027+0000 7f8af2e918c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-24T18:18:26.024 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:26.027+0000 7f8af2e918c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-24T18:18:26.025 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:26.027+0000 7f8af2e918c0 -1 bdev(0x563917aa7800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-24T18:18:26.025 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:26.027+0000 7f8af2e918c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-24T18:18:28.290 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-24T18:18:28.344 DEBUG:teuthology.orchestra.run.trial020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:18:28.407 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:28.411+0000 7f18822998c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-24T18:18:28.408 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:28.411+0000 7f18822998c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-24T18:18:28.408 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:28.411+0000 7f18822998c0 -1 bdev(0x557800263800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-24T18:18:28.408 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:28.411+0000 7f18822998c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-24T18:18:30.714 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-24T18:18:30.773 DEBUG:teuthology.orchestra.run.trial020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:18:30.843 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:30.843+0000 7fa5332438c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-24T18:18:30.843 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:30.847+0000 7fa5332438c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-24T18:18:30.844 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:30.847+0000 7fa5332438c0 -1 bdev(0x55cc956cb800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-24T18:18:30.844 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T18:18:30.847+0000 7fa5332438c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-24T18:18:33.117 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-24T18:18:33.171 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-24T18:18:33.172 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:33.172 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-24T18:18:33.228 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:33.228 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-24T18:18:33.280 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:33.280 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-24T18:18:33.332 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:33.332 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-24T18:18:33.384 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:33.384 DEBUG:teuthology.orchestra.run.trial020:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-24T18:18:33.431 INFO:tasks.ceph:Adding keys to all mons... 2026-02-24T18:18:33.431 DEBUG:teuthology.orchestra.run.trial020:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-24T18:18:33.481 INFO:teuthology.orchestra.run.trial020.stdout:[mgr.x] 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout: key = AQBv651pOkWPIhAAwIrbVwli29qzTFXWDyIf4w== 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout:[osd.0] 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout: key = AQBy651p963cARAAcjMRZdScFr9pfIfdIEmYww== 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout:[osd.1] 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout: key = AQB0651p0eK4GBAAmqf98n88Il9jo7x4rg8hgA== 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout:[osd.2] 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout: key = AQB2651pgyivMhAA7jogIQZt3P70m0Af4rj3RA== 2026-02-24T18:18:33.482 INFO:teuthology.orchestra.run.trial020.stdout:[client.0] 2026-02-24T18:18:33.483 INFO:teuthology.orchestra.run.trial020.stdout: key = AQBv651pPcETJhAAsqLBg1V8ly8OQ7dd8U+D4Q== 2026-02-24T18:18:33.484 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-24T18:18:33.548 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:18:33.612 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:18:33.675 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T18:18:33.739 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-24T18:18:33.804 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-24T18:18:33.805 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-24T18:18:33.855 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-24T18:18:33.924 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-24T18:18:33.974 DEBUG:teuthology.orchestra.run.trial020:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-24T18:18:34.018 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-24T18:18:34.019 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-24T18:18:34.019 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-24T18:18:34.060 INFO:tasks.ceph.mon.a:Started 2026-02-24T18:18:34.060 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph config set mgr mgr_pool false --force 2026-02-24T18:18:34.190 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-24T18:18:34.190 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-24T18:18:34.190 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-24T18:18:34.192 INFO:tasks.ceph.mgr.x:Started 2026-02-24T18:18:34.192 INFO:tasks.ceph:Setting crush tunables to default 2026-02-24T18:18:34.192 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd crush tunables default 2026-02-24T18:18:34.295 INFO:teuthology.orchestra.run.trial020.stderr:adjusted tunables profile to default 2026-02-24T18:18:34.298 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.299+0000 7fbe29d28280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-24T18:18:34.304 INFO:tasks.ceph:check_enable_crimson: False 2026-02-24T18:18:34.304 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-24T18:18:34.305 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:34.305 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-24T18:18:34.314 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:34.314 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-24T18:18:34.325 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.327+0000 7fbe29d28280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-24T18:18:34.333 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:18:34.333 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-24T18:18:34.362 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.363+0000 7fbe29d28280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-24T18:18:34.370 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd new 41405e39-39b2-4123-b663-1283aad326f2 0 2026-02-24T18:18:34.449 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.451+0000 7fbe29d28280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-24T18:18:34.505 INFO:teuthology.orchestra.run.trial020.stdout:0 2026-02-24T18:18:34.517 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd new 4827b0fa-0f0a-4731-b219-2bafdbf7bac8 1 2026-02-24T18:18:34.530 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.531+0000 7fbe29d28280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-24T18:18:34.567 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.571+0000 7fbe29d28280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-24T18:18:34.629 INFO:teuthology.orchestra.run.trial020.stdout:1 2026-02-24T18:18:34.638 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd new 2fbc4bb6-353f-4de0-a383-c0c6c207f765 2 2026-02-24T18:18:34.641 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.643+0000 7fbe29d28280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-24T18:18:34.738 INFO:teuthology.orchestra.run.trial020.stdout:2 2026-02-24T18:18:34.747 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-24T18:18:34.747 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-24T18:18:34.749 INFO:tasks.ceph.osd.0:Started 2026-02-24T18:18:34.749 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-24T18:18:34.749 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-24T18:18:34.817 INFO:tasks.ceph.osd.1:Started 2026-02-24T18:18:34.817 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-24T18:18:34.817 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-24T18:18:34.818 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:34.819+0000 7fbe29d28280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-24T18:18:34.819 INFO:tasks.ceph.osd.2:Started 2026-02-24T18:18:34.820 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:18:34.909 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:34.909 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":5,"fsid":"135e2b35-ac06-4607-a4ba-6aa45d264e7b","created":"2026-02-24T18:18:34.147883+0000","modified":"2026-02-24T18:18:34.744762+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T18:18:34.744762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"41405e39-39b2-4123-b663-1283aad326f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4827b0fa-0f0a-4731-b219-2bafdbf7bac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fbc4bb6-353f-4de0-a383-c0c6c207f765","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:18:34.919 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-24T18:18:34.919 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-24T18:18:35.123 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.127+0000 7fbe29d28280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-24T18:18:35.216 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.219+0000 7fbe29d28280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-24T18:18:35.220 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T18:18:35.243 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.247+0000 7fbe29d28280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-24T18:18:35.275 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.275+0000 7fbe29d28280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-24T18:18:35.308 INFO:teuthology.misc.health.trial020.stdout: 2026-02-24T18:18:35.308 INFO:teuthology.misc.health.trial020.stdout:{"epoch":5,"fsid":"135e2b35-ac06-4607-a4ba-6aa45d264e7b","created":"2026-02-24T18:18:34.147883+0000","modified":"2026-02-24T18:18:34.744762+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T18:18:34.744762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"41405e39-39b2-4123-b663-1283aad326f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4827b0fa-0f0a-4731-b219-2bafdbf7bac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fbc4bb6-353f-4de0-a383-c0c6c207f765","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:18:35.316 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T18:18:35.318 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.319+0000 7fbe29d28280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-24T18:18:35.422 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.423+0000 7fbe29d28280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-24T18:18:35.617 INFO:tasks.ceph.mgr.x.trial020.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-24T18:18:35.792 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.795+0000 7fbe29d28280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-24T18:18:35.824 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.827+0000 7fbe29d28280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-24T18:18:35.919 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.919+0000 7fbe29d28280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-24T18:18:35.943 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.947+0000 7fbe29d28280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-24T18:18:35.971 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.971+0000 7fbe29d28280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-24T18:18:35.998 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:35.999+0000 7fbe29d28280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-24T18:18:36.120 INFO:tasks.ceph.mgr.x.trial020.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T18:18:36.121 INFO:tasks.ceph.mgr.x.trial020.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T18:18:36.121 INFO:tasks.ceph.mgr.x.trial020.stderr: from numpy import show_config as show_numpy_config 2026-02-24T18:18:36.127 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:36.131+0000 7fbe29d28280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-24T18:18:36.154 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:36.155+0000 7fbe29d28280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-24T18:18:36.193 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:36.195+0000 7fbe29d28280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-24T18:18:36.328 INFO:tasks.ceph.osd.1.trial020.stderr:2026-02-24T18:18:36.331+0000 7f56f977d8c0 -1 Falling back to public interface 2026-02-24T18:18:36.372 INFO:tasks.ceph.osd.2.trial020.stderr:2026-02-24T18:18:36.375+0000 7f60d6d018c0 -1 Falling back to public interface 2026-02-24T18:18:36.812 INFO:tasks.ceph.osd.0.trial020.stderr:2026-02-24T18:18:36.815+0000 7f50b009a8c0 -1 Falling back to public interface 2026-02-24T18:18:37.219 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T18:18:37.219+0000 7fbd48579640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-24T18:18:39.324 INFO:tasks.ceph.osd.1.trial020.stderr:2026-02-24T18:18:39.327+0000 7f56f977d8c0 -1 osd.1 0 log_to_monitors true 2026-02-24T18:18:39.400 INFO:tasks.ceph.osd.2.trial020.stderr:2026-02-24T18:18:39.403+0000 7f60d6d018c0 -1 osd.2 0 log_to_monitors true 2026-02-24T18:18:40.055 INFO:tasks.ceph.osd.0.trial020.stderr:2026-02-24T18:18:40.059+0000 7f50b009a8c0 -1 osd.0 0 log_to_monitors true 2026-02-24T18:18:41.222 INFO:tasks.ceph.osd.0.trial020.stderr:2026-02-24T18:18:41.223+0000 7f50ac043640 -1 osd.0 0 waiting for initial osdmap 2026-02-24T18:18:41.223 INFO:tasks.ceph.osd.2.trial020.stderr:2026-02-24T18:18:41.227+0000 7f60d2caa640 -1 osd.2 0 waiting for initial osdmap 2026-02-24T18:18:41.224 INFO:tasks.ceph.osd.1.trial020.stderr:2026-02-24T18:18:41.227+0000 7f56f5726640 -1 osd.1 0 waiting for initial osdmap 2026-02-24T18:18:41.244 INFO:tasks.ceph.osd.0.trial020.stderr:2026-02-24T18:18:41.247+0000 7f50a7e7e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:18:41.248 INFO:tasks.ceph.osd.2.trial020.stderr:2026-02-24T18:18:41.251+0000 7f60ceae5640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:18:41.260 INFO:tasks.ceph.osd.1.trial020.stderr:2026-02-24T18:18:41.263+0000 7f56f1561640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-24T18:18:41.619 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T18:18:41.761 INFO:teuthology.misc.health.trial020.stdout: 2026-02-24T18:18:41.761 INFO:teuthology.misc.health.trial020.stdout:{"epoch":7,"fsid":"135e2b35-ac06-4607-a4ba-6aa45d264e7b","created":"2026-02-24T18:18:34.147883+0000","modified":"2026-02-24T18:18:41.224729+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T18:18:34.744762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"41405e39-39b2-4123-b663-1283aad326f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4827b0fa-0f0a-4731-b219-2bafdbf7bac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2fbc4bb6-353f-4de0-a383-c0c6c207f765","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:18:41.771 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T18:18:48.074 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T18:18:48.214 INFO:teuthology.misc.health.trial020.stdout: 2026-02-24T18:18:48.215 INFO:teuthology.misc.health.trial020.stdout:{"epoch":8,"fsid":"135e2b35-ac06-4607-a4ba-6aa45d264e7b","created":"2026-02-24T18:18:34.147883+0000","modified":"2026-02-24T18:18:42.227208+0000","last_up_change":"2026-02-24T18:18:42.227208+0000","last_in_change":"2026-02-24T18:18:34.744762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"41405e39-39b2-4123-b663-1283aad326f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6818","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6819","nonce":1817351360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6820","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6821","nonce":1817351360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6824","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6825","nonce":1817351360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6822","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6823","nonce":1817351360}]},"public_addr":"10.20.193.20:6819/1817351360","cluster_addr":"10.20.193.20:6821/1817351360","heartbeat_back_addr":"10.20.193.20:6825/1817351360","heartbeat_front_addr":"10.20.193.20:6823/1817351360","state":["exists","up"]},{"osd":1,"uuid":"4827b0fa-0f0a-4731-b219-2bafdbf7bac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6802","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6803","nonce":1077990978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6804","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6805","nonce":1077990978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6808","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6809","nonce":1077990978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6806","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6807","nonce":1077990978}]},"public_addr":"10.20.193.20:6803/1077990978","cluster_addr":"10.20.193.20:6805/1077990978","heartbeat_back_addr":"10.20.193.20:6809/1077990978","heartbeat_front_addr":"10.20.193.20:6807/1077990978","state":["exists","up"]},{"osd":2,"uuid":"2fbc4bb6-353f-4de0-a383-c0c6c207f765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6810","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6811","nonce":2532477871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6812","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6813","nonce":2532477871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6816","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6817","nonce":2532477871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6814","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6815","nonce":2532477871}]},"public_addr":"10.20.193.20:6811/2532477871","cluster_addr":"10.20.193.20:6813/2532477871","heartbeat_back_addr":"10.20.193.20:6817/2532477871","heartbeat_front_addr":"10.20.193.20:6815/2532477871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:18:48.224 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-24T18:18:48.224 INFO:tasks.ceph:Creating RBD pool 2026-02-24T18:18:48.224 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-24T18:18:49.233 INFO:teuthology.orchestra.run.trial020.stderr:pool 'rbd' created 2026-02-24T18:18:49.246 DEBUG:teuthology.orchestra.run.trial020:> rbd --cluster ceph pool init rbd 2026-02-24T18:18:52.250 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-24T18:18:52.250 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-24T18:18:52.250 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-24T18:18:52.251 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-24T18:18:52.251 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-24T18:18:52.416 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:52.426 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6800","nonce":1347351659},{"type":"v1","addr":"10.20.193.20:6801","nonce":1347351659}]},"active_addr":"10.20.193.20:6801/1347351659","active_change":"2026-02-24T18:18:36.219344+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.20:0","nonce":227575781}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.20:0","nonce":4005286799}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.20:0","nonce":3742970305}]}]} 2026-02-24T18:18:52.427 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-24T18:18:52.427 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-24T18:18:52.427 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:18:52.571 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:52.571 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":12,"fsid":"135e2b35-ac06-4607-a4ba-6aa45d264e7b","created":"2026-02-24T18:18:34.147883+0000","modified":"2026-02-24T18:18:52.247125+0000","last_up_change":"2026-02-24T18:18:42.227208+0000","last_in_change":"2026-02-24T18:18:34.744762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-24T18:18:48.384496+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41405e39-39b2-4123-b663-1283aad326f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6818","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6819","nonce":1817351360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6820","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6821","nonce":1817351360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6824","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6825","nonce":1817351360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6822","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6823","nonce":1817351360}]},"public_addr":"10.20.193.20:6819/1817351360","cluster_addr":"10.20.193.20:6821/1817351360","heartbeat_back_addr":"10.20.193.20:6825/1817351360","heartbeat_front_addr":"10.20.193.20:6823/1817351360","state":["exists","up"]},{"osd":1,"uuid":"4827b0fa-0f0a-4731-b219-2bafdbf7bac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6802","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6803","nonce":1077990978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6804","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6805","nonce":1077990978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6808","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6809","nonce":1077990978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6806","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6807","nonce":1077990978}]},"public_addr":"10.20.193.20:6803/1077990978","cluster_addr":"10.20.193.20:6805/1077990978","heartbeat_back_addr":"10.20.193.20:6809/1077990978","heartbeat_front_addr":"10.20.193.20:6807/1077990978","state":["exists","up"]},{"osd":2,"uuid":"2fbc4bb6-353f-4de0-a383-c0c6c207f765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6810","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6811","nonce":2532477871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6812","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6813","nonce":2532477871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6816","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6817","nonce":2532477871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6814","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6815","nonce":2532477871}]},"public_addr":"10.20.193.20:6811/2532477871","cluster_addr":"10.20.193.20:6813/2532477871","heartbeat_back_addr":"10.20.193.20:6817/2532477871","heartbeat_front_addr":"10.20.193.20:6815/2532477871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:18:52.581 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-24T18:18:52.581 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:18:52.728 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:52.728 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":12,"fsid":"135e2b35-ac06-4607-a4ba-6aa45d264e7b","created":"2026-02-24T18:18:34.147883+0000","modified":"2026-02-24T18:18:52.247125+0000","last_up_change":"2026-02-24T18:18:42.227208+0000","last_in_change":"2026-02-24T18:18:34.744762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-24T18:18:48.384496+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41405e39-39b2-4123-b663-1283aad326f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6818","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6819","nonce":1817351360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6820","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6821","nonce":1817351360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6824","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6825","nonce":1817351360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6822","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6823","nonce":1817351360}]},"public_addr":"10.20.193.20:6819/1817351360","cluster_addr":"10.20.193.20:6821/1817351360","heartbeat_back_addr":"10.20.193.20:6825/1817351360","heartbeat_front_addr":"10.20.193.20:6823/1817351360","state":["exists","up"]},{"osd":1,"uuid":"4827b0fa-0f0a-4731-b219-2bafdbf7bac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6802","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6803","nonce":1077990978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6804","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6805","nonce":1077990978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6808","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6809","nonce":1077990978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6806","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6807","nonce":1077990978}]},"public_addr":"10.20.193.20:6803/1077990978","cluster_addr":"10.20.193.20:6805/1077990978","heartbeat_back_addr":"10.20.193.20:6809/1077990978","heartbeat_front_addr":"10.20.193.20:6807/1077990978","state":["exists","up"]},{"osd":2,"uuid":"2fbc4bb6-353f-4de0-a383-c0c6c207f765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6810","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6811","nonce":2532477871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6812","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6813","nonce":2532477871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6816","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6817","nonce":2532477871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6814","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6815","nonce":2532477871}]},"public_addr":"10.20.193.20:6811/2532477871","cluster_addr":"10.20.193.20:6813/2532477871","heartbeat_back_addr":"10.20.193.20:6817/2532477871","heartbeat_front_addr":"10.20.193.20:6815/2532477871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:18:52.739 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-24T18:18:52.740 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-24T18:18:52.740 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-24T18:18:52.823 INFO:teuthology.orchestra.run.trial020.stdout:34359738373 2026-02-24T18:18:52.824 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T18:18:52.825 INFO:teuthology.orchestra.run.trial020.stdout:34359738372 2026-02-24T18:18:52.826 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T18:18:52.827 INFO:teuthology.orchestra.run.trial020.stdout:34359738372 2026-02-24T18:18:52.827 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T18:18:53.013 INFO:teuthology.orchestra.run.trial020.stdout:34359738371 2026-02-24T18:18:53.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2026-02-24T18:18:53.038 INFO:teuthology.orchestra.run.trial020.stdout:34359738371 2026-02-24T18:18:53.044 INFO:teuthology.orchestra.run.trial020.stdout:34359738372 2026-02-24T18:18:53.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2026-02-24T18:18:53.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2026-02-24T18:18:54.025 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T18:18:54.051 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T18:18:54.058 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T18:18:54.171 INFO:teuthology.orchestra.run.trial020.stdout:34359738371 2026-02-24T18:18:54.183 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2026-02-24T18:18:54.217 INFO:teuthology.orchestra.run.trial020.stdout:34359738372 2026-02-24T18:18:54.219 INFO:teuthology.orchestra.run.trial020.stdout:34359738371 2026-02-24T18:18:54.228 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2026-02-24T18:18:54.232 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2026-02-24T18:18:55.184 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T18:18:55.230 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T18:18:55.232 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T18:18:55.328 INFO:teuthology.orchestra.run.trial020.stdout:34359738373 2026-02-24T18:18:55.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.1 2026-02-24T18:18:55.340 DEBUG:teuthology.parallel:result is None 2026-02-24T18:18:55.373 INFO:teuthology.orchestra.run.trial020.stdout:34359738374 2026-02-24T18:18:55.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738374 for osd.0 2026-02-24T18:18:55.384 DEBUG:teuthology.parallel:result is None 2026-02-24T18:18:55.438 INFO:teuthology.orchestra.run.trial020.stdout:34359738373 2026-02-24T18:18:55.449 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.2 2026-02-24T18:18:55.449 DEBUG:teuthology.parallel:result is None 2026-02-24T18:18:55.449 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T18:18:55.449 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:18:55.604 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:55.605 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T18:18:55.615 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-24T18:18:54.223555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81212,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491588,"statfs":{"total":322122547200,"available":322039386112,"internally_reserved":0,"allocated":327680,"data_stored":81620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253698+0000","last_change":"2026-02-24T18:18:52.253698+0000","last_active":"2026-02-24T18:18:52.253698+0000","last_peered":"2026-02-24T18:18:52.253698+0000","last_clean":"2026-02-24T18:18:52.253698+0000","last_became_active":"2026-02-24T18:18:50.249066+0000","last_became_peered":"2026-02-24T18:18:50.249066+0000","last_unstale":"2026-02-24T18:18:52.253698+0000","last_undegraded":"2026-02-24T18:18:52.253698+0000","last_fullsized":"2026-02-24T18:18:52.253698+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:46:05.765794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070762199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253607+0000","last_change":"2026-02-24T18:18:52.253607+0000","last_active":"2026-02-24T18:18:52.253607+0000","last_peered":"2026-02-24T18:18:52.253607+0000","last_clean":"2026-02-24T18:18:52.253607+0000","last_became_active":"2026-02-24T18:18:50.249405+0000","last_became_peered":"2026-02-24T18:18:50.249405+0000","last_unstale":"2026-02-24T18:18:52.253607+0000","last_undegraded":"2026-02-24T18:18:52.253607+0000","last_fullsized":"2026-02-24T18:18:52.253607+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:36:13.285373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064563700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.250241+0000","last_change":"2026-02-24T18:18:52.250241+0000","last_active":"2026-02-24T18:18:52.250241+0000","last_peered":"2026-02-24T18:18:52.250241+0000","last_clean":"2026-02-24T18:18:52.250241+0000","last_became_active":"2026-02-24T18:18:50.249055+0000","last_became_peered":"2026-02-24T18:18:50.249055+0000","last_unstale":"2026-02-24T18:18:52.250241+0000","last_undegraded":"2026-02-24T18:18:52.250241+0000","last_fullsized":"2026-02-24T18:18:52.250241+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:21:31.791793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033314800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253562+0000","last_change":"2026-02-24T18:18:52.253562+0000","last_active":"2026-02-24T18:18:52.253562+0000","last_peered":"2026-02-24T18:18:52.253562+0000","last_clean":"2026-02-24T18:18:52.253562+0000","last_became_active":"2026-02-24T18:18:50.249327+0000","last_became_peered":"2026-02-24T18:18:50.249327+0000","last_unstale":"2026-02-24T18:18:52.253562+0000","last_undegraded":"2026-02-24T18:18:52.253562+0000","last_fullsized":"2026-02-24T18:18:52.253562+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:37:29.642093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065132700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":20,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253485+0000","last_change":"2026-02-24T18:18:52.253485+0000","last_active":"2026-02-24T18:18:52.253485+0000","last_peered":"2026-02-24T18:18:52.253485+0000","last_clean":"2026-02-24T18:18:52.253485+0000","last_became_active":"2026-02-24T18:18:50.249028+0000","last_became_peered":"2026-02-24T18:18:50.249028+0000","last_unstale":"2026-02-24T18:18:52.253485+0000","last_undegraded":"2026-02-24T18:18:52.253485+0000","last_fullsized":"2026-02-24T18:18:52.253485+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:49:57.350554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057687899999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.252449+0000","last_change":"2026-02-24T18:18:52.252032+0000","last_active":"2026-02-24T18:18:52.252449+0000","last_peered":"2026-02-24T18:18:52.252449+0000","last_clean":"2026-02-24T18:18:52.252449+0000","last_became_active":"2026-02-24T18:18:50.248784+0000","last_became_peered":"2026-02-24T18:18:50.248784+0000","last_unstale":"2026-02-24T18:18:52.252449+0000","last_undegraded":"2026-02-24T18:18:52.252449+0000","last_fullsized":"2026-02-24T18:18:52.252449+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:02:07.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064190000000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.250180+0000","last_change":"2026-02-24T18:18:52.250180+0000","last_active":"2026-02-24T18:18:52.250180+0000","last_peered":"2026-02-24T18:18:52.250180+0000","last_clean":"2026-02-24T18:18:52.250180+0000","last_became_active":"2026-02-24T18:18:50.249075+0000","last_became_peered":"2026-02-24T18:18:50.249075+0000","last_unstale":"2026-02-24T18:18:52.250180+0000","last_undegraded":"2026-02-24T18:18:52.250180+0000","last_fullsized":"2026-02-24T18:18:52.250180+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:02:59.019103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028864500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253414+0000","last_change":"2026-02-24T18:18:52.253414+0000","last_active":"2026-02-24T18:18:52.253414+0000","last_peered":"2026-02-24T18:18:52.253414+0000","last_clean":"2026-02-24T18:18:52.253414+0000","last_became_active":"2026-02-24T18:18:50.249535+0000","last_became_peered":"2026-02-24T18:18:50.249535+0000","last_unstale":"2026-02-24T18:18:52.253414+0000","last_undegraded":"2026-02-24T18:18:52.253414+0000","last_fullsized":"2026-02-24T18:18:52.253414+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:08:53.720990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055293499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27068,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830532,"statfs":{"total":107374182400,"available":107346464768,"internally_reserved":0,"allocated":106496,"data_stored":27194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:18:55.616 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:18:55.767 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:55.768 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T18:18:55.779 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-24T18:18:54.223555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81212,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491588,"statfs":{"total":322122547200,"available":322039386112,"internally_reserved":0,"allocated":327680,"data_stored":81620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253698+0000","last_change":"2026-02-24T18:18:52.253698+0000","last_active":"2026-02-24T18:18:52.253698+0000","last_peered":"2026-02-24T18:18:52.253698+0000","last_clean":"2026-02-24T18:18:52.253698+0000","last_became_active":"2026-02-24T18:18:50.249066+0000","last_became_peered":"2026-02-24T18:18:50.249066+0000","last_unstale":"2026-02-24T18:18:52.253698+0000","last_undegraded":"2026-02-24T18:18:52.253698+0000","last_fullsized":"2026-02-24T18:18:52.253698+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:46:05.765794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070762199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253607+0000","last_change":"2026-02-24T18:18:52.253607+0000","last_active":"2026-02-24T18:18:52.253607+0000","last_peered":"2026-02-24T18:18:52.253607+0000","last_clean":"2026-02-24T18:18:52.253607+0000","last_became_active":"2026-02-24T18:18:50.249405+0000","last_became_peered":"2026-02-24T18:18:50.249405+0000","last_unstale":"2026-02-24T18:18:52.253607+0000","last_undegraded":"2026-02-24T18:18:52.253607+0000","last_fullsized":"2026-02-24T18:18:52.253607+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:36:13.285373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064563700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.250241+0000","last_change":"2026-02-24T18:18:52.250241+0000","last_active":"2026-02-24T18:18:52.250241+0000","last_peered":"2026-02-24T18:18:52.250241+0000","last_clean":"2026-02-24T18:18:52.250241+0000","last_became_active":"2026-02-24T18:18:50.249055+0000","last_became_peered":"2026-02-24T18:18:50.249055+0000","last_unstale":"2026-02-24T18:18:52.250241+0000","last_undegraded":"2026-02-24T18:18:52.250241+0000","last_fullsized":"2026-02-24T18:18:52.250241+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:21:31.791793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033314800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253562+0000","last_change":"2026-02-24T18:18:52.253562+0000","last_active":"2026-02-24T18:18:52.253562+0000","last_peered":"2026-02-24T18:18:52.253562+0000","last_clean":"2026-02-24T18:18:52.253562+0000","last_became_active":"2026-02-24T18:18:50.249327+0000","last_became_peered":"2026-02-24T18:18:50.249327+0000","last_unstale":"2026-02-24T18:18:52.253562+0000","last_undegraded":"2026-02-24T18:18:52.253562+0000","last_fullsized":"2026-02-24T18:18:52.253562+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:37:29.642093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065132700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":20,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253485+0000","last_change":"2026-02-24T18:18:52.253485+0000","last_active":"2026-02-24T18:18:52.253485+0000","last_peered":"2026-02-24T18:18:52.253485+0000","last_clean":"2026-02-24T18:18:52.253485+0000","last_became_active":"2026-02-24T18:18:50.249028+0000","last_became_peered":"2026-02-24T18:18:50.249028+0000","last_unstale":"2026-02-24T18:18:52.253485+0000","last_undegraded":"2026-02-24T18:18:52.253485+0000","last_fullsized":"2026-02-24T18:18:52.253485+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:49:57.350554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057687899999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":21,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.252449+0000","last_change":"2026-02-24T18:18:52.252032+0000","last_active":"2026-02-24T18:18:52.252449+0000","last_peered":"2026-02-24T18:18:52.252449+0000","last_clean":"2026-02-24T18:18:52.252449+0000","last_became_active":"2026-02-24T18:18:50.248784+0000","last_became_peered":"2026-02-24T18:18:50.248784+0000","last_unstale":"2026-02-24T18:18:52.252449+0000","last_undegraded":"2026-02-24T18:18:52.252449+0000","last_fullsized":"2026-02-24T18:18:52.252449+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:02:07.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064190000000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.250180+0000","last_change":"2026-02-24T18:18:52.250180+0000","last_active":"2026-02-24T18:18:52.250180+0000","last_peered":"2026-02-24T18:18:52.250180+0000","last_clean":"2026-02-24T18:18:52.250180+0000","last_became_active":"2026-02-24T18:18:50.249075+0000","last_became_peered":"2026-02-24T18:18:50.249075+0000","last_unstale":"2026-02-24T18:18:52.250180+0000","last_undegraded":"2026-02-24T18:18:52.250180+0000","last_fullsized":"2026-02-24T18:18:52.250180+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:02:59.019103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028864500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-02-24T18:18:52.253414+0000","last_change":"2026-02-24T18:18:52.253414+0000","last_active":"2026-02-24T18:18:52.253414+0000","last_peered":"2026-02-24T18:18:52.253414+0000","last_clean":"2026-02-24T18:18:52.253414+0000","last_became_active":"2026-02-24T18:18:50.249535+0000","last_became_peered":"2026-02-24T18:18:50.249535+0000","last_unstale":"2026-02-24T18:18:52.253414+0000","last_undegraded":"2026-02-24T18:18:52.253414+0000","last_fullsized":"2026-02-24T18:18:52.253414+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:08:53.720990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055293499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27068,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830532,"statfs":{"total":107374182400,"available":107346464768,"internally_reserved":0,"allocated":106496,"data_stored":27194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":110592,"data_stored":27213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:18:55.780 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T18:18:55.780 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-24T18:18:55.780 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-24T18:18:55.781 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-24T18:18:55.948 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:18:55.948 INFO:teuthology.orchestra.run.trial020.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-24T18:18:55.958 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-24T18:18:55.958 INFO:teuthology.run_tasks:Running task workunit... 2026-02-24T18:18:55.992 INFO:tasks.workunit:Pulling workunits from ref f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-24T18:18:55.993 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-24T18:18:55.993 INFO:tasks.workunit:timeout=3h 2026-02-24T18:18:55.993 INFO:tasks.workunit:cleanup=True 2026-02-24T18:18:55.993 DEBUG:teuthology.orchestra.run.trial020:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-24T18:18:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:18:56.007 INFO:teuthology.orchestra.run.trial020.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-24T18:18:56.007 DEBUG:teuthology.orchestra.run.trial020:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-24T18:18:56.055 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-24T18:18:56.055 DEBUG:teuthology.orchestra.run.trial020:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-24T18:18:56.103 DEBUG:teuthology.orchestra.run.trial020:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-24T18:18:56.154 INFO:tasks.workunit.client.0.trial020.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-24T18:19:18.276 INFO:tasks.workunit.client.0.trial020.stderr:Note: switching to 'f90f4eaebb28b619cf76e80cbbe997656da69579'. 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr:state without impacting any branches by switching back to a branch. 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:18.277 INFO:tasks.workunit.client.0.trial020.stderr: git switch -c 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr:Or undo this operation with: 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr: git switch - 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:18.278 INFO:tasks.workunit.client.0.trial020.stderr:HEAD is now at f90f4eaebb2 Merge branch 'wip-74712-squid' of https://github.com/idryomov/ceph into wip-yuri6-testing-2026-02-22-1629-squid 2026-02-24T18:19:18.286 DEBUG:teuthology.orchestra.run.trial020:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-24T18:19:18.332 INFO:tasks.workunit.client.0.trial020.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-24T18:19:18.333 INFO:tasks.workunit.client.0.trial020.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T18:19:18.333 INFO:tasks.workunit.client.0.trial020.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-24T18:19:18.428 INFO:tasks.workunit.client.0.trial020.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-24T18:19:18.459 INFO:tasks.workunit.client.0.trial020.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-24T18:19:18.485 INFO:tasks.workunit.client.0.trial020.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-24T18:19:18.486 INFO:tasks.workunit.client.0.trial020.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T18:19:18.486 INFO:tasks.workunit.client.0.trial020.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-24T18:19:18.508 INFO:tasks.workunit.client.0.trial020.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-24T18:19:18.512 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T18:19:18.512 DEBUG:teuthology.orchestra.run.trial020:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-24T18:19:18.560 INFO:tasks.workunit:Running workunits matching cephtool/test_kvstore_tool.sh on client.0... 2026-02-24T18:19:18.562 INFO:tasks.workunit:Running workunit cephtool/test_kvstore_tool.sh... 2026-02-24T18:19:18.563 DEBUG:teuthology.orchestra.run.trial020:workunit test cephtool/test_kvstore_tool.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh 2026-02-24T18:19:18.610 INFO:tasks.workunit.client.0.trial020.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ TIMEOUT=300 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ MAX_TIMEOUT=15 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ PG_NUM=4 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ TMPDIR=/tmp 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-24T18:19:18.612 INFO:tasks.workunit.client.0.trial020.stderr:++ type xmlstarlet 2026-02-24T18:19:18.613 INFO:tasks.workunit.client.0.trial020.stderr:++ XMLSTARLET=xmlstarlet 2026-02-24T18:19:18.613 INFO:tasks.workunit.client.0.trial020.stderr:+++ uname 2026-02-24T18:19:18.613 INFO:tasks.workunit.client.0.trial020.stderr:++ '[' Linux = FreeBSD ']' 2026-02-24T18:19:18.614 INFO:tasks.workunit.client.0.trial020.stderr:++ SED=sed 2026-02-24T18:19:18.614 INFO:tasks.workunit.client.0.trial020.stderr:++ AWK=awk 2026-02-24T18:19:18.614 INFO:tasks.workunit.client.0.trial020.stderr:+++ stty -a 2026-02-24T18:19:18.614 INFO:tasks.workunit.client.0.trial020.stderr:+++ head -1 2026-02-24T18:19:18.614 INFO:tasks.workunit.client.0.trial020.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-24T18:19:18.615 INFO:tasks.workunit.client.0.trial020.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-24T18:19:18.615 INFO:tasks.workunit.client.0.trial020.stderr:++ termwidth= 2026-02-24T18:19:18.615 INFO:tasks.workunit.client.0.trial020.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-24T18:19:18.615 INFO:tasks.workunit.client.0.trial020.stderr:++ DIFFCOLOPTS='-y ' 2026-02-24T18:19:18.615 INFO:tasks.workunit.client.0.trial020.stderr:++ KERNCORE=kernel.core_pattern 2026-02-24T18:19:18.615 INFO:tasks.workunit.client.0.trial020.stderr:++ EXTRA_OPTS= 2026-02-24T18:19:18.618 INFO:tasks.workunit.client.0.trial020.stderr:++ test '' = TESTS 2026-02-24T18:19:18.618 INFO:tasks.workunit.client.0.trial020.stderr:+ set -e 2026-02-24T18:19:18.618 INFO:tasks.workunit.client.0.trial020.stderr:+ set -o functrace 2026-02-24T18:19:18.618 INFO:tasks.workunit.client.0.trial020.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-24T18:19:18.618 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:10: : SUDO=sudo 2026-02-24T18:19:18.618 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:11: : export CEPH_DEV=1 2026-02-24T18:19:18.618 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:11: : CEPH_DEV=1 2026-02-24T18:19:18.619 INFO:tasks.workunit.client.0.trial020.stdout:note: test ceph_kvstore_tool with bluestore 2026-02-24T18:19:18.619 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:13: : echo note: test ceph_kvstore_tool with bluestore 2026-02-24T18:19:18.619 INFO:tasks.workunit.client.0.trial020.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:21: : mktemp -d ./cephtool.XXX 2026-02-24T18:19:18.620 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:21: : TEMP_DIR=./cephtool.ZWT 2026-02-24T18:19:18.620 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:22: : trap 'rm -fr ./cephtool.ZWT' 0 2026-02-24T18:19:18.620 INFO:tasks.workunit.client.0.trial020.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:24: : mktemp ./cephtool.ZWT/test_invalid.XXX 2026-02-24T18:19:18.620 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:24: : TEMP_FILE=./cephtool.ZWT/test_invalid.qPm 2026-02-24T18:19:18.621 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:69: : test_ceph_kvstore_tool 2026-02-24T18:19:18.621 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:29: test_ceph_kvstore_tool: ceph-objectstore-tool --data-path ./cephtool.ZWT --op mkfs --no-mon-config 2026-02-24T18:19:20.026 INFO:tasks.workunit.client.0.trial020.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:32: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT list 2026-02-24T18:19:20.026 INFO:tasks.workunit.client.0.trial020.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:32: test_ceph_kvstore_tool: wc -l 2026-02-24T18:19:21.128 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:32: test_ceph_kvstore_tool: origin_kv_nums=11 2026-02-24T18:19:21.129 INFO:tasks.workunit.client.0.trial020.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT list 2026-02-24T18:19:21.129 INFO:tasks.workunit.client.0.trial020.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: head -n 1 2026-02-24T18:19:21.129 INFO:tasks.workunit.client.0.trial020.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: awk '{print $1}' 2026-02-24T18:19:22.219 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:35: test_ceph_kvstore_tool: prefix=B 2026-02-24T18:19:22.220 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:36: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT exists B 2026-02-24T18:19:22.233 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.234+0000 7f53c18ecbc0 1 bdev(0x55f44e56b800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:22.233 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.234+0000 7f53c18ecbc0 1 bdev(0x55f44e56b800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:22.233 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.234+0000 7f53c18ecbc0 1 bdev(0x55f44e56b800 ./cephtool.ZWT/block) close 2026-02-24T18:19:22.523 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bdev(0x55f44e56b800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:22.523 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bdev(0x55f44e56b800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:22.524 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:22.524 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bdev(0x55f44e56bc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:22.525 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bdev(0x55f44e56bc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:22.525 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:22.525 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs mount 2026-02-24T18:19:22.525 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:22.526 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.526 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.527 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.527 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.527 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.527 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:22.527 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.526+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:22.535 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.534+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:22.535 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.534+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:22.535 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.534+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:22.535 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.534+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:22.535 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.534+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:22.536 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 1 freelist init 2026-02-24T18:19:22.536 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 1 freelist _read_cfg 2026-02-24T18:19:22.536 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-24T18:19:22.536 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 0 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-24T18:19:22.536 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000222 seconds, extent_count=1 2026-02-24T18:19:22.536 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:22.536 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 1 bluefs umount 2026-02-24T18:19:22.537 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.538+0000 7f53c18ecbc0 1 bdev(0x55f44e56bc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:22.803 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bdev(0x55f44e56bc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:22.803 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bdev(0x55f44e56bc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:22.804 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:22.804 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs mount 2026-02-24T18:19:22.804 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:22.804 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.804 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.805 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.805 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.805 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:22.805 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-24T18:19:22.805 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.806+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:22.812 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.814+0000 7f53c18ecbc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:22.813 INFO:tasks.workunit.client.0.trial020.stdout:(B, ) exists 2026-02-24T18:19:22.813 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.814+0000 7f53c18ecbc0 1 bluefs umount 2026-02-24T18:19:22.813 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:22.814+0000 7f53c18ecbc0 1 bdev(0x55f44e56bc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:23.083 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.086+0000 7f53c18ecbc0 1 freelist shutdown 2026-02-24T18:19:23.083 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.086+0000 7f53c18ecbc0 1 bdev(0x55f44e56b800 ./cephtool.ZWT/block) close 2026-02-24T18:19:23.306 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:37: test_ceph_kvstore_tool: expect_false ceph-kvstore-tool bluestore-kv ./cephtool.ZWT exists Bnotexist 2026-02-24T18:19:23.306 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:17: expect_false: set -x 2026-02-24T18:19:23.306 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT exists Bnotexist 2026-02-24T18:19:23.320 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.322+0000 7f31e7177bc0 1 bdev(0x556c73453800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:23.321 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.322+0000 7f31e7177bc0 1 bdev(0x556c73453800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:23.321 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.322+0000 7f31e7177bc0 1 bdev(0x556c73453800 ./cephtool.ZWT/block) close 2026-02-24T18:19:23.607 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bdev(0x556c73453800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:23.607 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bdev(0x556c73453800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:23.608 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:23.608 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bdev(0x556c73453c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:23.609 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bdev(0x556c73453c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:23.609 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:23.609 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs mount 2026-02-24T18:19:23.609 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:23.610 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.610 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.611 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.611 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.611 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.611 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:23.611 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.610+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:23.616 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:23.616 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:23.616 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:23.617 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:23.617 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 freelist init 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 freelist _read_cfg 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 0 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000112 seconds, extent_count=1 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bluefs umount 2026-02-24T18:19:23.618 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.618+0000 7f31e7177bc0 1 bdev(0x556c73453c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:23.887 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bdev(0x556c73453c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:23.888 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bdev(0x556c73453c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:23.888 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:23.888 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs mount 2026-02-24T18:19:23.888 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:23.888 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.889 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.889 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.889 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.889 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:23.889 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-24T18:19:23.889 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.890+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:23.896 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.898+0000 7f31e7177bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:23.897 INFO:tasks.workunit.client.0.trial020.stdout:(Bnotexist, ) does not exist 2026-02-24T18:19:23.897 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.898+0000 7f31e7177bc0 1 bluefs umount 2026-02-24T18:19:23.897 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:23.898+0000 7f31e7177bc0 1 bdev(0x556c73453c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:24.171 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.174+0000 7f31e7177bc0 1 freelist shutdown 2026-02-24T18:19:24.171 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.174+0000 7f31e7177bc0 1 bdev(0x556c73453800 ./cephtool.ZWT/block) close 2026-02-24T18:19:24.391 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: return 0 2026-02-24T18:19:24.392 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:40: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT list-crc 2026-02-24T18:19:24.405 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.406+0000 7f89cdff8bc0 1 bdev(0x55ddc634f800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:24.405 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.406+0000 7f89cdff8bc0 1 bdev(0x55ddc634f800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:24.405 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.406+0000 7f89cdff8bc0 1 bdev(0x55ddc634f800 ./cephtool.ZWT/block) close 2026-02-24T18:19:24.687 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bdev(0x55ddc634f800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:24.688 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bdev(0x55ddc634f800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:24.688 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:24.688 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bdev(0x55ddc634fc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:24.688 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bdev(0x55ddc634fc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:24.688 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:24.688 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs mount 2026-02-24T18:19:24.688 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:24.690 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.690 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.690 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.690 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.690 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.691 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:24.691 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.690+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:24.697 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:24.697 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:24.697 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:24.698 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:24.698 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 freelist init 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 freelist _read_cfg 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 0 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000198 seconds, extent_count=1 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bluefs umount 2026-02-24T18:19:24.699 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.698+0000 7f89cdff8bc0 1 bdev(0x55ddc634fc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:24.975 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bdev(0x55ddc634fc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:24.976 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bdev(0x55ddc634fc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:24.976 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:24.976 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs mount 2026-02-24T18:19:24.976 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:24.977 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.977 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.977 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.977 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.977 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:24.977 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-24T18:19:24.977 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.978+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:24.984 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.986+0000 7f89cdff8bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:24.985 INFO:tasks.workunit.client.0.trial020.stdout:B blocks 3887034139 2026-02-24T18:19:24.985 INFO:tasks.workunit.client.0.trial020.stdout:B blocks_per_key 2356384685 2026-02-24T18:19:24.985 INFO:tasks.workunit.client.0.trial020.stdout:B bytes_per_block 1745006751 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:B size 714193852 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:S blobid_max 3417961130 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:S freelist_type 753695201 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:S min_alloc_size 1458662255 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:S min_compat_ondisk_format 2796427507 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:S nid_max 1692495819 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:S ondisk_format 3256356326 2026-02-24T18:19:24.986 INFO:tasks.workunit.client.0.trial020.stdout:S per_pool_omap 490065073 2026-02-24T18:19:24.987 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.986+0000 7f89cdff8bc0 1 bluefs umount 2026-02-24T18:19:24.987 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:24.986+0000 7f89cdff8bc0 1 bdev(0x55ddc634fc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:25.255 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.258+0000 7f89cdff8bc0 1 freelist shutdown 2026-02-24T18:19:25.255 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.258+0000 7f89cdff8bc0 1 bdev(0x55ddc634f800 ./cephtool.ZWT/block) close 2026-02-24T18:19:25.471 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:41: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT list-crc B 2026-02-24T18:19:25.483 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.486+0000 7f41d7b4cbc0 1 bdev(0x555e5d72f800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:25.484 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.486+0000 7f41d7b4cbc0 1 bdev(0x555e5d72f800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:25.484 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.486+0000 7f41d7b4cbc0 1 bdev(0x555e5d72f800 ./cephtool.ZWT/block) close 2026-02-24T18:19:25.772 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bdev(0x555e5d72f800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:25.772 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bdev(0x555e5d72f800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:25.772 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:25.772 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bdev(0x555e5d72fc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:25.773 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bdev(0x555e5d72fc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:25.773 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:25.773 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs mount 2026-02-24T18:19:25.773 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:25.774 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:25.775 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:25.775 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:25.775 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:25.775 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:25.775 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:25.775 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.774+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:25.783 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.782+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:25.783 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.782+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:25.783 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.782+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:25.783 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.782+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:25.783 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.782+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:25.784 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 1 freelist init 2026-02-24T18:19:25.784 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 1 freelist _read_cfg 2026-02-24T18:19:25.784 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-24T18:19:25.785 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 0 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-24T18:19:25.785 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000207 seconds, extent_count=1 2026-02-24T18:19:25.785 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:25.785 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 1 bluefs umount 2026-02-24T18:19:25.785 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:25.786+0000 7f41d7b4cbc0 1 bdev(0x555e5d72fc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:26.051 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bdev(0x555e5d72fc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:26.052 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bdev(0x555e5d72fc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:26.052 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:26.052 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs mount 2026-02-24T18:19:26.052 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:26.053 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.053 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.053 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.053 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.053 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.053 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-24T18:19:26.053 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.054+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:26.061 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.062+0000 7f41d7b4cbc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:26.061 INFO:tasks.workunit.client.0.trial020.stdout:B blocks 3887034139 2026-02-24T18:19:26.061 INFO:tasks.workunit.client.0.trial020.stdout:B blocks_per_key 2356384685 2026-02-24T18:19:26.061 INFO:tasks.workunit.client.0.trial020.stdout:B bytes_per_block 1745006751 2026-02-24T18:19:26.062 INFO:tasks.workunit.client.0.trial020.stdout:B size 714193852 2026-02-24T18:19:26.062 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.062+0000 7f41d7b4cbc0 1 bluefs umount 2026-02-24T18:19:26.062 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.062+0000 7f41d7b4cbc0 1 bdev(0x555e5d72fc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:26.331 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.334+0000 7f41d7b4cbc0 1 freelist shutdown 2026-02-24T18:19:26.331 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.334+0000 7f41d7b4cbc0 1 bdev(0x555e5d72f800 ./cephtool.ZWT/block) close 2026-02-24T18:19:26.555 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:44: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT list B 2026-02-24T18:19:26.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.570+0000 7f3e45385bc0 1 bdev(0x558941b6b800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:26.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.570+0000 7f3e45385bc0 1 bdev(0x558941b6b800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:26.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.570+0000 7f3e45385bc0 1 bdev(0x558941b6b800 ./cephtool.ZWT/block) close 2026-02-24T18:19:26.852 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bdev(0x558941b6b800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:26.852 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bdev(0x558941b6b800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:26.852 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:26.852 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bdev(0x558941b6bc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:26.852 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bdev(0x558941b6bc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:26.853 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:26.853 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs mount 2026-02-24T18:19:26.853 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:26.854 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:26.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:26.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.854+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:26.860 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:26.860 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:26.860 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:26.860 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:26.860 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:26.861 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 freelist init 2026-02-24T18:19:26.861 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 freelist _read_cfg 2026-02-24T18:19:26.862 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-24T18:19:26.862 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 0 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-24T18:19:26.862 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000132 seconds, extent_count=1 2026-02-24T18:19:26.862 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:26.862 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bluefs umount 2026-02-24T18:19:26.862 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:26.862+0000 7f3e45385bc0 1 bdev(0x558941b6bc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:27.131 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bdev(0x558941b6bc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:27.132 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bdev(0x558941b6bc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:27.132 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:27.132 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs mount 2026-02-24T18:19:27.132 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:27.133 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.133 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.133 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.133 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.133 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.133 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-24T18:19:27.133 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.134+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:27.141 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.142+0000 7f3e45385bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:27.141 INFO:tasks.workunit.client.0.trial020.stdout:B blocks 2026-02-24T18:19:27.141 INFO:tasks.workunit.client.0.trial020.stdout:B blocks_per_key 2026-02-24T18:19:27.141 INFO:tasks.workunit.client.0.trial020.stdout:B bytes_per_block 2026-02-24T18:19:27.142 INFO:tasks.workunit.client.0.trial020.stdout:B size 2026-02-24T18:19:27.142 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.142+0000 7f3e45385bc0 1 bluefs umount 2026-02-24T18:19:27.142 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.142+0000 7f3e45385bc0 1 bdev(0x558941b6bc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:27.419 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.422+0000 7f3e45385bc0 1 freelist shutdown 2026-02-24T18:19:27.419 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.422+0000 7f3e45385bc0 1 bdev(0x558941b6b800 ./cephtool.ZWT/block) close 2026-02-24T18:19:27.635 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:47: test_ceph_kvstore_tool: echo helloworld 2026-02-24T18:19:27.635 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:48: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT set TESTPREFIX TESTKEY in ./cephtool.ZWT/test_invalid.qPm 2026-02-24T18:19:27.649 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.650+0000 7f0f70c4abc0 1 bdev(0x55c791f31800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:27.649 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.650+0000 7f0f70c4abc0 1 bdev(0x55c791f31800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:27.649 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.650+0000 7f0f70c4abc0 1 bdev(0x55c791f31800 ./cephtool.ZWT/block) close 2026-02-24T18:19:27.932 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bdev(0x55c791f31800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:27.932 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bdev(0x55c791f31800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:27.932 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:27.932 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bdev(0x55c791f31c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:27.932 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bdev(0x55c791f31c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:27.932 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:27.933 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs mount 2026-02-24T18:19:27.933 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:27.934 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.934 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.935 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.935 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.935 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:27.935 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:27.935 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.934+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:27.940 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:27.940 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:27.940 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:27.940 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:27.940 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:27.941 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 freelist init 2026-02-24T18:19:27.941 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 freelist _read_cfg 2026-02-24T18:19:27.941 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2026-02-24T18:19:27.941 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 0 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ... 2026-02-24T18:19:27.942 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 0.000135 seconds, extent_count=1 2026-02-24T18:19:27.942 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:27.942 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bluefs umount 2026-02-24T18:19:27.942 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:27.942+0000 7f0f70c4abc0 1 bdev(0x55c791f31c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:28.211 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bdev(0x55c791f31c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:28.212 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bdev(0x55c791f31c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:28.212 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:28.212 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs mount 2026-02-24T18:19:28.212 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:28.213 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:28.213 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:28.213 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:28.213 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:28.213 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:28.213 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluefs mount shared_bdev_used = 4653056 2026-02-24T18:19:28.213 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.214+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:28.225 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.226+0000 7f0f70c4abc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:28.229 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.230+0000 7f0f70c4abc0 1 bluefs umount 2026-02-24T18:19:28.229 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.230+0000 7f0f70c4abc0 1 bdev(0x55c791f31c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:28.519 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.522+0000 7f0f70c4abc0 1 freelist shutdown 2026-02-24T18:19:28.519 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.522+0000 7f0f70c4abc0 1 bdev(0x55c791f31800 ./cephtool.ZWT/block) close 2026-02-24T18:19:28.715 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:49: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT exists TESTPREFIX TESTKEY 2026-02-24T18:19:28.729 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.730+0000 7fc571a09bc0 1 bdev(0x5615049bb800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:28.729 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.730+0000 7fc571a09bc0 1 bdev(0x5615049bb800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:28.730 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:28.730+0000 7fc571a09bc0 1 bdev(0x5615049bb800 ./cephtool.ZWT/block) close 2026-02-24T18:19:29.011 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bdev(0x5615049bb800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:29.012 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bdev(0x5615049bb800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:29.012 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:29.012 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bdev(0x5615049bbc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:29.013 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bdev(0x5615049bbc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:29.013 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:29.013 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs mount 2026-02-24T18:19:29.013 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:29.015 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.015 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.015 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.015 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.015 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.015 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:29.015 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.014+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:29.020 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:29.020 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:29.021 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:29.021 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:29.021 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:29.021 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 freelist init 2026-02-24T18:19:29.021 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 freelist _read_cfg 2026-02-24T18:19:29.022 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:29.022 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bluefs umount 2026-02-24T18:19:29.022 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.022+0000 7fc571a09bc0 1 bdev(0x5615049bbc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:29.295 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bdev(0x5615049bbc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:29.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bdev(0x5615049bbc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:29.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:29.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs mount 2026-02-24T18:19:29.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:29.297 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.297 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.297 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.297 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.297 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:29.297 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-24T18:19:29.297 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.298+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:29.301 INFO:tasks.workunit.client.0.trial020.stdout:(TESTPREFIX, TESTKEY) exists 2026-02-24T18:19:29.302 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.302+0000 7fc571a09bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:29.302 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.302+0000 7fc571a09bc0 1 bluefs umount 2026-02-24T18:19:29.302 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.302+0000 7fc571a09bc0 1 bdev(0x5615049bbc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:29.587 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.590+0000 7fc571a09bc0 1 freelist shutdown 2026-02-24T18:19:29.587 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.590+0000 7fc571a09bc0 1 bdev(0x5615049bb800 ./cephtool.ZWT/block) close 2026-02-24T18:19:29.801 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:52: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT get TESTPREFIX TESTKEY out ./cephtool.ZWT/test_invalid.qPm.bak 2026-02-24T18:19:29.816 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.818+0000 7fb9e741ebc0 1 bdev(0x561ccff6b800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:29.816 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.818+0000 7fb9e741ebc0 1 bdev(0x561ccff6b800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:29.816 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:29.818+0000 7fb9e741ebc0 1 bdev(0x561ccff6b800 ./cephtool.ZWT/block) close 2026-02-24T18:19:30.096 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bdev(0x561ccff6b800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:30.096 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bdev(0x561ccff6b800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:30.096 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:30.096 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bdev(0x561ccff6bc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:30.097 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bdev(0x561ccff6bc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:30.097 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:30.097 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs mount 2026-02-24T18:19:30.097 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:30.099 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.099 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.099 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.099 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.099 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.099 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:30.099 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.098+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:30.105 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:30.105 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:30.105 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:30.105 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:30.105 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:30.105 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 freelist init 2026-02-24T18:19:30.105 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 freelist _read_cfg 2026-02-24T18:19:30.106 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:30.106 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bluefs umount 2026-02-24T18:19:30.106 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.106+0000 7fb9e741ebc0 1 bdev(0x561ccff6bc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:30.379 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bdev(0x561ccff6bc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:30.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bdev(0x561ccff6bc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:30.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:30.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs mount 2026-02-24T18:19:30.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:30.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:30.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-24T18:19:30.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.382+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:30.385 INFO:tasks.workunit.client.0.trial020.stdout:(TESTPREFIX, TESTKEY) 2026-02-24T18:19:30.386 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.386+0000 7fb9e741ebc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:30.386 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.386+0000 7fb9e741ebc0 1 bluefs umount 2026-02-24T18:19:30.386 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.386+0000 7fb9e741ebc0 1 bdev(0x561ccff6bc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:30.659 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.662+0000 7fb9e741ebc0 1 freelist shutdown 2026-02-24T18:19:30.659 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.662+0000 7fb9e741ebc0 1 bdev(0x561ccff6b800 ./cephtool.ZWT/block) close 2026-02-24T18:19:30.879 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:53: test_ceph_kvstore_tool: diff ./cephtool.ZWT/test_invalid.qPm ./cephtool.ZWT/test_invalid.qPm.bak 2026-02-24T18:19:30.881 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:56: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT rm TESTPREFIX TESTKEY 2026-02-24T18:19:30.901 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.902+0000 7f8aa8cb2bc0 1 bdev(0x563c70575800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:30.901 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.902+0000 7f8aa8cb2bc0 1 bdev(0x563c70575800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:30.901 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:30.902+0000 7f8aa8cb2bc0 1 bdev(0x563c70575800 ./cephtool.ZWT/block) close 2026-02-24T18:19:31.191 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bdev(0x563c70575800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:31.192 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bdev(0x563c70575800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:31.192 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:31.192 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bdev(0x563c70575c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:31.192 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bdev(0x563c70575c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:31.193 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:31.193 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs mount 2026-02-24T18:19:31.193 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:31.194 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.194 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.194 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.194 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.194 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.195 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:31.195 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.194+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:31.201 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:31.202 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:31.202 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:31.202 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:31.202 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:31.202 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 freelist init 2026-02-24T18:19:31.202 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 freelist _read_cfg 2026-02-24T18:19:31.203 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:31.203 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bluefs umount 2026-02-24T18:19:31.203 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.202+0000 7f8aa8cb2bc0 1 bdev(0x563c70575c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:31.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bdev(0x563c70575c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:31.480 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bdev(0x563c70575c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:31.480 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:31.480 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs mount 2026-02-24T18:19:31.480 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:31.481 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.481 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.481 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.481 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.481 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:31.481 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-24T18:19:31.481 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.482+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:31.490 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.490+0000 7f8aa8cb2bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:31.492 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.494+0000 7f8aa8cb2bc0 1 bluefs umount 2026-02-24T18:19:31.492 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.494+0000 7f8aa8cb2bc0 1 bdev(0x563c70575c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:31.771 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.774+0000 7f8aa8cb2bc0 1 freelist shutdown 2026-02-24T18:19:31.771 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.774+0000 7f8aa8cb2bc0 1 bdev(0x563c70575800 ./cephtool.ZWT/block) close 2026-02-24T18:19:31.983 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:57: test_ceph_kvstore_tool: expect_false ceph-kvstore-tool bluestore-kv ./cephtool.ZWT exists TESTPREFIX TESTKEY 2026-02-24T18:19:31.983 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:17: expect_false: set -x 2026-02-24T18:19:31.983 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT exists TESTPREFIX TESTKEY 2026-02-24T18:19:31.996 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.998+0000 7faf9e728bc0 1 bdev(0x55c33a307800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:31.996 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.998+0000 7faf9e728bc0 1 bdev(0x55c33a307800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:31.996 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:31.998+0000 7faf9e728bc0 1 bdev(0x55c33a307800 ./cephtool.ZWT/block) close 2026-02-24T18:19:32.288 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bdev(0x55c33a307800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:32.288 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bdev(0x55c33a307800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:32.288 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:32.288 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bdev(0x55c33a307c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:32.289 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bdev(0x55c33a307c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:32.289 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:32.289 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs mount 2026-02-24T18:19:32.289 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:32.290 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.290 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.290 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.290 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.290 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.290 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:32.291 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.290+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:32.295 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.294+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:32.295 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.294+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:32.295 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.294+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:32.295 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.294+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:32.295 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.294+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:32.295 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.298+0000 7faf9e728bc0 1 freelist init 2026-02-24T18:19:32.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.298+0000 7faf9e728bc0 1 freelist _read_cfg 2026-02-24T18:19:32.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.298+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:32.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.298+0000 7faf9e728bc0 1 bluefs umount 2026-02-24T18:19:32.296 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.298+0000 7faf9e728bc0 1 bdev(0x55c33a307c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:32.568 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bdev(0x55c33a307c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:32.568 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bdev(0x55c33a307c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:32.568 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:32.568 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs mount 2026-02-24T18:19:32.568 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:32.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:32.569 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluefs mount shared_bdev_used = 4980736 2026-02-24T18:19:32.570 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.570+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:32.575 INFO:tasks.workunit.client.0.trial020.stdout:(TESTPREFIX, TESTKEY) does not exist 2026-02-24T18:19:32.575 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.574+0000 7faf9e728bc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:32.576 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.574+0000 7faf9e728bc0 1 bluefs umount 2026-02-24T18:19:32.576 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.574+0000 7faf9e728bc0 1 bdev(0x55c33a307c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:32.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.858+0000 7faf9e728bc0 1 freelist shutdown 2026-02-24T18:19:32.855 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:32.858+0000 7faf9e728bc0 1 bdev(0x55c33a307800 ./cephtool.ZWT/block) close 2026-02-24T18:19:33.073 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:18: expect_false: return 0 2026-02-24T18:19:33.074 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:60: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT compact 2026-02-24T18:19:33.087 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.086+0000 7f28e6e6abc0 1 bdev(0x56520fdfb800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:33.087 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.086+0000 7f28e6e6abc0 1 bdev(0x56520fdfb800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:33.087 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.086+0000 7f28e6e6abc0 1 bdev(0x56520fdfb800 ./cephtool.ZWT/block) close 2026-02-24T18:19:33.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bdev(0x56520fdfb800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:33.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bdev(0x56520fdfb800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:33.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:33.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bdev(0x56520fdfbc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:33.380 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bdev(0x56520fdfbc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:33.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:33.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs mount 2026-02-24T18:19:33.381 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:33.383 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.383 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.383 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.383 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.383 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.383 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:33.383 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.382+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:33.388 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:33.389 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:33.389 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:33.389 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:33.389 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:33.389 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 freelist init 2026-02-24T18:19:33.389 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 freelist _read_cfg 2026-02-24T18:19:33.390 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:33.390 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bluefs umount 2026-02-24T18:19:33.390 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.390+0000 7f28e6e6abc0 1 bdev(0x56520fdfbc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:33.659 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bdev(0x56520fdfbc00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:33.660 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bdev(0x56520fdfbc00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:33.660 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:33.660 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs mount 2026-02-24T18:19:33.660 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:33.661 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.661 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.661 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.661 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.661 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:33.661 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluefs mount shared_bdev_used = 4980736 2026-02-24T18:19:33.661 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.662+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:33.670 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.670+0000 7f28e6e6abc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:33.670 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.670+0000 7f28e6e6abc0 2 rocksdb: compact starting 2026-02-24T18:19:33.673 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.674+0000 7f28e6e6abc0 2 rocksdb: compact completed 2026-02-24T18:19:33.676 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.678+0000 7f28e6e6abc0 1 bluefs umount 2026-02-24T18:19:33.676 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.678+0000 7f28e6e6abc0 1 bdev(0x56520fdfbc00 ./cephtool.ZWT/block) close 2026-02-24T18:19:33.963 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.966+0000 7f28e6e6abc0 1 freelist shutdown 2026-02-24T18:19:33.963 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:33.966+0000 7f28e6e6abc0 1 bdev(0x56520fdfb800 ./cephtool.ZWT/block) close 2026-02-24T18:19:34.172 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:63: test_ceph_kvstore_tool: ceph-kvstore-tool bluestore-kv ./cephtool.ZWT destructive-repair 2026-02-24T18:19:34.185 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.186 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.186 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) close 2026-02-24T18:19:34.476 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.476 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.476 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:34.476 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.476 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.477 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:34.477 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount 2026-02-24T18:19:34.477 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:34.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:34.479 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:34.484 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:34.484 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:34.485 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:34.485 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:34.485 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:34.486 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 freelist init 2026-02-24T18:19:34.486 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 freelist _read_cfg 2026-02-24T18:19:34.486 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:34.486 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluefs umount 2026-02-24T18:19:34.486 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:34.755 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.756 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.756 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:34.756 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount 2026-02-24T18:19:34.756 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:34.757 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.757 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.757 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.757 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.757 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.757 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-24T18:19:34.757 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:34.779 INFO:tasks.workunit.client.0.trial020.stdout:destructive-repair completed without reporting an error 2026-02-24T18:19:34.788 INFO:tasks.workunit.client.0.trial020.stderr:*** Caught signal (Segmentation fault) ** 2026-02-24T18:19:34.788 INFO:tasks.workunit.client.0.trial020.stderr: in thread 7f878734fbc0 thread_name:ceph-kvstore-to 2026-02-24T18:19:34.788 INFO:tasks.workunit.client.0.trial020.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-24T18:19:34.788 INFO:tasks.workunit.client.0.trial020.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f878855a520] 2026-02-24T18:19:34.788 INFO:tasks.workunit.client.0.trial020.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x556008b2efa4] 2026-02-24T18:19:34.788 INFO:tasks.workunit.client.0.trial020.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x556008b2e8d9] 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 4: (BlueStore::_close_db()+0x31c) [0x556008980e9c] 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 5: (BlueStore::umount()+0x239) [0x55600898b469] 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55600892ced3] 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 7: main() 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f8788541d90] 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 9: __libc_start_main() 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 10: _start() 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr:2026-02-24T18:19:34.782+0000 7f878734fbc0 -1 *** Caught signal (Segmentation fault) ** 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: in thread 7f878734fbc0 thread_name:ceph-kvstore-to 2026-02-24T18:19:34.789 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f878855a520] 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x556008b2efa4] 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x556008b2e8d9] 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 4: (BlueStore::_close_db()+0x31c) [0x556008980e9c] 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 5: (BlueStore::umount()+0x239) [0x55600898b469] 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55600892ced3] 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 7: main() 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f8788541d90] 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 9: __libc_start_main() 2026-02-24T18:19:34.790 INFO:tasks.workunit.client.0.trial020.stderr: 10: _start() 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr:--- begin dump of recent events --- 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -190> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command assert hook 0x55604053a280 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -189> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command abort hook 0x55604053a280 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -188> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command leak_some_memory hook 0x55604053a280 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -187> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perfcounters_dump hook 0x55604053a280 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -186> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command 1 hook 0x55604053a280 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -185> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf dump hook 0x55604053a280 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -184> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perfcounters_schema hook 0x55604053a280 2026-02-24T18:19:34.791 INFO:tasks.workunit.client.0.trial020.stderr: -183> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf histogram dump hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -182> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command 2 hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -181> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf schema hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -180> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command counter dump hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -179> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command counter schema hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -178> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf histogram schema hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -177> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf reset hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -176> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config show hook 0x55604053a280 2026-02-24T18:19:34.792 INFO:tasks.workunit.client.0.trial020.stderr: -175> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config help hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -174> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config set hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -173> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config unset hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -172> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config get hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -171> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config diff hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -170> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config diff get hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -169> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command injectargs hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -168> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command log flush hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -167> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command log dump hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -166> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command log reopen hook 0x55604053a280 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -165> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command dump_mempools hook 0x556040626068 2026-02-24T18:19:34.793 INFO:tasks.workunit.client.0.trial020.stderr: -164> 2026-02-24T18:19:34.186+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _open_db_and_around::NCB::read_only=0, to_repair=1 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -163> 2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -162> 2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -161> 2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) close 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -160> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -159> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -158> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -157> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore bluefs device info hook 0x55604053a2f0 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -156> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs stats hook 0x55604053a2f0 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -155> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs files list hook 0x55604053a2f0 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -154> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs debug_inject_read_zeros hook 0x55604053a2f0 2026-02-24T18:19:34.794 INFO:tasks.workunit.client.0.trial020.stderr: -153> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -152> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -151> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -150> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -149> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 bluefs mount super: super(uuid 12826af7-1d0d-4b3b-9959-6fcefc6aa589 osd a663c4cd-00d1-458f-bf65-6d658806b8b2 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-24T18:19:33.681761+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -148> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -147> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -146> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -145> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -144> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -143> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.795 INFO:tasks.workunit.client.0.trial020.stderr: -142> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -141> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -140> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -139> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -138> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -137> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -136> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -135> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -134> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -133> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -132> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -131> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.796 INFO:tasks.workunit.client.0.trial020.stderr: -130> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -129> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -128> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -127> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -126> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -125> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -124> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -123> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -122> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -121> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -120> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -119> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -118> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.797 INFO:tasks.workunit.client.0.trial020.stderr: -117> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -116> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -115> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -114> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -113> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _open_super_meta::NCB::freelist_type=null 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -112> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -111> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -110> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -109> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor snappy 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -108> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor snappy 0x5560408a5530 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -107> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0x5560408a5530 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -106> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor snappy loaded and registered 2026-02-24T18:19:34.798 INFO:tasks.workunit.client.0.trial020.stderr: -105> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0x5560408a5530 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -104> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zlib = 0 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -103> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor zlib 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -102> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor zlib 0x5560408a5470 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -101> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zlib = 0x5560408a5470 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -100> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor zlib loaded and registered 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -99> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zlib = 0x5560408a5470 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -98> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zstd = 0 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -97> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor zstd 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -96> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor zstd 0x5560409a4030 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -95> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zstd = 0x5560409a4030 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -94> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor zstd loaded and registered 2026-02-24T18:19:34.799 INFO:tasks.workunit.client.0.trial020.stderr: -93> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zstd = 0x5560409a4030 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -92> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor lz4 = 0 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -91> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor lz4 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -90> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor lz4 0x5560409a40c0 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -89> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor lz4 = 0x5560409a40c0 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -88> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor lz4 loaded and registered 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -87> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor lz4 = 0x5560409a40c0 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -86> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0x5560408a5530 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -85> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _open_fm::NCB::freelist_type=null 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -84> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 freelist init 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -83> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 freelist _read_cfg 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -82> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator dump block hook 0x5560409a40f0 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -81> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator score block hook 0x5560409a40f0 2026-02-24T18:19:34.800 INFO:tasks.workunit.client.0.trial020.stderr: -80> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation block hook 0x5560409a40f0 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -79> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation histogram block hook 0x5560409a40f0 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -78> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator dump recovery hook 0x5560409a41b0 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -77> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator score recovery hook 0x5560409a41b0 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -76> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation recovery hook 0x5560409a41b0 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -75> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation histogram recovery hook 0x5560409a41b0 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -74> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::__restore_allocator::file_size=132,sizeof(extent_t)=16 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -73> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::__restore_allocator::READ--extent_count=1, read_alloc_size= 107374174208, file_size=132 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -72> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::__restore_allocator::READ duration=0.000408 seconds, s_serial=3 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -71> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::restore_allocator:: calling copy_allocator(bitmap_allocator -> shared_alloc.a) 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -70> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::copy_allocator::count num_entries=1 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -69> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::copy_allocator::copy num_entries=1 2026-02-24T18:19:34.801 INFO:tasks.workunit.client.0.trial020.stderr: -68> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::restore_allocator::restored in 0.000655 seconds, num_entries=1 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -67> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator dump recovery 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -66> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator fragmentation histogram recovery 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -65> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator fragmentation recovery 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -64> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator score recovery 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -63> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() completed successfully alloc=0x55604092f200 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -62> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -61> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluefs umount 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -60> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluefs debug_inject_read_zeros 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -59> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluefs files list 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -58> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluefs stats 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -57> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore bluefs device info 2026-02-24T18:19:34.802 INFO:tasks.workunit.client.0.trial020.stderr: -56> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -55> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore bluefs device info hook 0x55604053ab90 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -54> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs stats hook 0x55604053ab90 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -53> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs files list hook 0x55604053ab90 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -52> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs debug_inject_read_zeros hook 0x55604053ab90 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -51> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -50> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -49> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -48> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -47> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 bluefs mount super: super(uuid 12826af7-1d0d-4b3b-9959-6fcefc6aa589 osd a663c4cd-00d1-458f-bf65-6d658806b8b2 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-24T18:19:33.681761+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -46> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -45> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.803 INFO:tasks.workunit.client.0.trial020.stderr: -44> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -43> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -42> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -41> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -40> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -39> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -38> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -37> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -36> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.804 INFO:tasks.workunit.client.0.trial020.stderr: -35> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -34> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -33> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -32> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -31> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -30> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -29> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -28> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.805 INFO:tasks.workunit.client.0.trial020.stderr: -27> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -26> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 bluestore::NCB::invalidate_allocation_file_on_bluefs::invalidate using bluefs->truncate(p_handle, 0) 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -25> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -24> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -23> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -22> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -21> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -20> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.806 INFO:tasks.workunit.client.0.trial020.stderr: -19> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -18> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -17> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -16> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -15> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -14> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -13> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -12> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.807 INFO:tasks.workunit.client.0.trial020.stderr: -11> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -10> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -9> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -8> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -7> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -6> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -5> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -4> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -3> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -2> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.808 INFO:tasks.workunit.client.0.trial020.stderr: -1> 2026-02-24T18:19:34.782+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) umount 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 0> 2026-02-24T18:19:34.782+0000 7f878734fbc0 -1 *** Caught signal (Segmentation fault) ** 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: in thread 7f878734fbc0 thread_name:ceph-kvstore-to 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f878855a520] 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x556008b2efa4] 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x556008b2e8d9] 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 4: (BlueStore::_close_db()+0x31c) [0x556008980e9c] 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 5: (BlueStore::umount()+0x239) [0x55600898b469] 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55600892ced3] 2026-02-24T18:19:34.809 INFO:tasks.workunit.client.0.trial020.stderr: 7: main() 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f8788541d90] 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 9: __libc_start_main() 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 10: _start() 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr:--- logging levels --- 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 none 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 lockdep 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 context 2026-02-24T18:19:34.810 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 crush 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_balancer 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_locker 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_log 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_log_expire 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_migrator 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 3/ 5 mds_quiesce 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 buffer 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 timer 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 filer 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 striper 2026-02-24T18:19:34.811 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 objecter 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rados 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd_mirror 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd_replay 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd_pwl 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 journaler 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 objectcacher 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 immutable_obj_cache 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 client 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 osd 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 optracker 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 objclass 2026-02-24T18:19:34.812 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 3 filestore 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 3 journal 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 0 ms 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mon 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 0/10 monc 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 paxos 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 tp 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 auth 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 crypto 2026-02-24T18:19:34.813 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 finisher 2026-02-24T18:19:34.814 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 reserver 2026-02-24T18:19:34.814 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 heartbeatmap 2026-02-24T18:19:34.814 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 perfcounter 2026-02-24T18:19:34.814 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw 2026-02-24T18:19:34.814 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_sync 2026-02-24T18:19:34.814 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_datacache 2026-02-24T18:19:34.814 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_access 2026-02-24T18:19:34.815 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_dbstore 2026-02-24T18:19:34.815 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_flight 2026-02-24T18:19:34.815 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_lifecycle 2026-02-24T18:19:34.815 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_notification 2026-02-24T18:19:34.815 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 javaclient 2026-02-24T18:19:34.815 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 asok 2026-02-24T18:19:34.815 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 throttle 2026-02-24T18:19:34.816 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 0 refs 2026-02-24T18:19:34.816 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 compressor 2026-02-24T18:19:34.816 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 bluestore 2026-02-24T18:19:34.816 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 bluefs 2026-02-24T18:19:34.816 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 3 bdev 2026-02-24T18:19:34.816 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 kstore 2026-02-24T18:19:34.816 INFO:tasks.workunit.client.0.trial020.stderr: 2/ 2 rocksdb 2026-02-24T18:19:34.817 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 fuse 2026-02-24T18:19:34.817 INFO:tasks.workunit.client.0.trial020.stderr: 2/ 5 mgr 2026-02-24T18:19:34.817 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mgrc 2026-02-24T18:19:34.817 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 dpdk 2026-02-24T18:19:34.817 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 eventtrace 2026-02-24T18:19:34.817 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 prioritycache 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 test 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 cephfs_mirror 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 cephsqlite 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 crimson_interrupt 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_onode 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_odata 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_omap 2026-02-24T18:19:34.818 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_tm 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_t 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_cleaner 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_epm 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_lba 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_fixedkv_tree 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_cache 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_journal 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_device 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_backref 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 alienstore 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mclock 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 cyanstore 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 ceph_exporter 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 memstore 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 trace 2026-02-24T18:19:34.819 INFO:tasks.workunit.client.0.trial020.stderr: -2/-2 (syslog threshold) 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: 99/99 (stderr threshold) 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr:--- pthread ID / name mapping for recent threads --- 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: 7f878734fbc0 / ceph-kvstore-to 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: max_recent 500 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: max_new 1000 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: log_file 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr:--- end dump of recent events --- 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr:--- begin dump of recent events --- 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: -190> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command assert hook 0x55604053a280 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: -189> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command abort hook 0x55604053a280 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: -188> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command leak_some_memory hook 0x55604053a280 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: -187> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perfcounters_dump hook 0x55604053a280 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: -186> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command 1 hook 0x55604053a280 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: -185> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf dump hook 0x55604053a280 2026-02-24T18:19:34.820 INFO:tasks.workunit.client.0.trial020.stderr: -184> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perfcounters_schema hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -183> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf histogram dump hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -182> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command 2 hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -181> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf schema hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -180> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command counter dump hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -179> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command counter schema hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -178> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf histogram schema hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -177> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command perf reset hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -176> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config show hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -175> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config help hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -174> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config set hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -173> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config unset hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -172> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config get hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -171> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config diff hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -170> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command config diff get hook 0x55604053a280 2026-02-24T18:19:34.821 INFO:tasks.workunit.client.0.trial020.stderr: -169> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command injectargs hook 0x55604053a280 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -168> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command log flush hook 0x55604053a280 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -167> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command log dump hook 0x55604053a280 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -166> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command log reopen hook 0x55604053a280 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -165> 2026-02-24T18:19:34.182+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command dump_mempools hook 0x556040626068 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -164> 2026-02-24T18:19:34.186+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _open_db_and_around::NCB::read_only=0, to_repair=1 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -163> 2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -162> 2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -161> 2026-02-24T18:19:34.186+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) close 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -160> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -159> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561800 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -158> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -157> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore bluefs device info hook 0x55604053a2f0 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -156> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs stats hook 0x55604053a2f0 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -155> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs files list hook 0x55604053a2f0 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -154> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs debug_inject_read_zeros hook 0x55604053a2f0 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -153> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.822 INFO:tasks.workunit.client.0.trial020.stderr: -152> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -151> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -150> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -149> 2026-02-24T18:19:34.478+0000 7f878734fbc0 5 bluefs mount super: super(uuid 12826af7-1d0d-4b3b-9959-6fcefc6aa589 osd a663c4cd-00d1-458f-bf65-6d658806b8b2 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-24T18:19:33.681761+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -148> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -147> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -146> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -145> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -144> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -143> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -142> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluefs mount shared_bdev_used = 0 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -141> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -140> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -139> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -138> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -137> 2026-02-24T18:19:34.478+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -136> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.823 INFO:tasks.workunit.client.0.trial020.stderr: -135> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -134> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -133> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -132> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -131> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -130> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -129> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -128> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -127> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -126> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -125> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -124> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -123> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -122> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -121> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -120> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -119> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -118> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.824 INFO:tasks.workunit.client.0.trial020.stderr: -117> 2026-02-24T18:19:34.482+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -116> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -115> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old nid_max 0 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -114> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta old blobid_max 0 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -113> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _open_super_meta::NCB::freelist_type=null 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -112> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -111> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _open_super_meta min_alloc_size 0x1000 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -110> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -109> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor snappy 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -108> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor snappy 0x5560408a5530 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -107> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0x5560408a5530 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -106> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor snappy loaded and registered 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -105> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0x5560408a5530 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -104> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zlib = 0 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -103> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor zlib 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -102> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor zlib 0x5560408a5470 2026-02-24T18:19:34.825 INFO:tasks.workunit.client.0.trial020.stderr: -101> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zlib = 0x5560408a5470 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -100> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor zlib loaded and registered 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -99> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zlib = 0x5560408a5470 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -98> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zstd = 0 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -97> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor zstd 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -96> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor zstd 0x5560409a4030 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -95> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zstd = 0x5560409a4030 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -94> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor zstd loaded and registered 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -93> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor zstd = 0x5560409a4030 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -92> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor lz4 = 0 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -91> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load compressor lz4 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -90> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 add compressor lz4 0x5560409a40c0 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -89> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor lz4 = 0x5560409a40c0 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -88> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 load: compressor lz4 loaded and registered 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -87> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor lz4 = 0x5560409a40c0 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -86> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 get compressor snappy = 0x5560408a5530 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -85> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _open_fm::NCB::freelist_type=null 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -84> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 freelist init 2026-02-24T18:19:34.826 INFO:tasks.workunit.client.0.trial020.stderr: -83> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 freelist _read_cfg 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -82> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator dump block hook 0x5560409a40f0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -81> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator score block hook 0x5560409a40f0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -80> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation block hook 0x5560409a40f0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -79> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation histogram block hook 0x5560409a40f0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -78> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator dump recovery hook 0x5560409a41b0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -77> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator score recovery hook 0x5560409a41b0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -76> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation recovery hook 0x5560409a41b0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -75> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore allocator fragmentation histogram recovery hook 0x5560409a41b0 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -74> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::__restore_allocator::file_size=132,sizeof(extent_t)=16 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -73> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::__restore_allocator::READ--extent_count=1, read_alloc_size= 107374174208, file_size=132 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -72> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::__restore_allocator::READ duration=0.000408 seconds, s_serial=3 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -71> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::restore_allocator:: calling copy_allocator(bitmap_allocator -> shared_alloc.a) 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -70> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::copy_allocator::count num_entries=1 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -69> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::copy_allocator::copy num_entries=1 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -68> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore::NCB::restore_allocator::restored in 0.000655 seconds, num_entries=1 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -67> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator dump recovery 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -66> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator fragmentation histogram recovery 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -65> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator fragmentation recovery 2026-02-24T18:19:34.827 INFO:tasks.workunit.client.0.trial020.stderr: -64> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore allocator score recovery 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -63> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) _init_alloc::NCB::restore_allocator() completed successfully alloc=0x55604092f200 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -62> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _init_alloc loaded 100 GiB in 1 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18ffffe000, fragmentation 0 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -61> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bluefs umount 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -60> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluefs debug_inject_read_zeros 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -59> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluefs files list 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -58> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluefs stats 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -57> 2026-02-24T18:19:34.486+0000 7f878734fbc0 5 asok(0x5560405c8000) unregister_commands bluestore bluefs device info 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -56> 2026-02-24T18:19:34.486+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) close 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -55> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluestore bluefs device info hook 0x55604053ab90 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -54> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs stats hook 0x55604053ab90 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -53> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs files list hook 0x55604053ab90 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -52> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 asok(0x5560405c8000) register_command bluefs debug_inject_read_zeros hook 0x55604053ab90 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -51> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open path ./cephtool.ZWT/block 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -50> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bdev(0x556040561c00 ./cephtool.ZWT/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -49> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs add_block_device bdev 1 path ./cephtool.ZWT/block size 100 GiB 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -48> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -47> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 bluefs mount super: super(uuid 12826af7-1d0d-4b3b-9959-6fcefc6aa589 osd a663c4cd-00d1-458f-bf65-6d658806b8b2 seq 4 block_size 0x1000 log_fnode 0xfile(ino 1 size 0x0 mtime 2026-02-24T18:19:33.681761+0000 allocated 10000 alloc_commit 10000 extents [1:0xc2000~10000])) 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -46> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -45> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.828 INFO:tasks.workunit.client.0.trial020.stderr: -44> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -43> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -42> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -41> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -40> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluefs mount shared_bdev_used = 4915200 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -39> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 bluestore(./cephtool.ZWT) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -38> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -37> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -36> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -35> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -34> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -33> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -32> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -31> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -30> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -29> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -28> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -27> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -26> 2026-02-24T18:19:34.758+0000 7f878734fbc0 5 bluestore::NCB::invalidate_allocation_file_on_bluefs::invalidate using bluefs->truncate(p_handle, 0) 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -25> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -24> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.829 INFO:tasks.workunit.client.0.trial020.stderr: -23> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -22> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -21> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -20> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -19> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -18> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -17> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -16> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -15> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -14> 2026-02-24T18:19:34.758+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -13> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option writable_file_max_buffer_size = 0 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -12> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_base = 1073741824 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -11> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option level0_file_num_compaction_trigger = 8 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -10> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_background_jobs = 4 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -9> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option write_buffer_size = 16777216 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -8> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option compaction_style = kCompactionStyleLevel 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -7> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option compaction_readahead_size = 2MB 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -6> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_bytes_for_level_multiplier = 8 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -5> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_write_buffer_number = 64 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -4> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option max_total_wal_size = 1073741824 2026-02-24T18:19:34.830 INFO:tasks.workunit.client.0.trial020.stderr: -3> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option min_write_buffer_number_to_merge = 6 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: -2> 2026-02-24T18:19:34.762+0000 7f878734fbc0 1 set rocksdb option compression = kLZ4Compression 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: -1> 2026-02-24T18:19:34.782+0000 7f878734fbc0 5 bluestore(./cephtool.ZWT) umount 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 0> 2026-02-24T18:19:34.782+0000 7f878734fbc0 -1 *** Caught signal (Segmentation fault) ** 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: in thread 7f878734fbc0 thread_name:ceph-kvstore-to 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable) 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f878855a520] 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x556008b2efa4] 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x556008b2e8d9] 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 4: (BlueStore::_close_db()+0x31c) [0x556008980e9c] 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 5: (BlueStore::umount()+0x239) [0x55600898b469] 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 6: ceph-kvstore-tool(+0x1f8ed3) [0x55600892ced3] 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 7: main() 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f8788541d90] 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 9: __libc_start_main() 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 10: _start() 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr:--- logging levels --- 2026-02-24T18:19:34.831 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 none 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 lockdep 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 context 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 crush 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_balancer 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_locker 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_log 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_log_expire 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mds_migrator 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 3/ 5 mds_quiesce 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 buffer 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 timer 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 filer 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 striper 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 1 objecter 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rados 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd_mirror 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd_replay 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 rbd_pwl 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 journaler 2026-02-24T18:19:34.832 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 objectcacher 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 immutable_obj_cache 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 client 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 osd 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 optracker 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 objclass 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 3 filestore 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 3 journal 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 0 ms 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mon 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 0/10 monc 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 paxos 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 tp 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 auth 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 crypto 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 finisher 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 reserver 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 heartbeatmap 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 perfcounter 2026-02-24T18:19:34.833 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_sync 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_datacache 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_access 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_dbstore 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_flight 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_lifecycle 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 rgw_notification 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 javaclient 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 asok 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 1 throttle 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 0 refs 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 compressor 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 bluestore 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 bluefs 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 3 bdev 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 kstore 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 2/ 2 rocksdb 2026-02-24T18:19:34.834 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 fuse 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 2/ 5 mgr 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mgrc 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 dpdk 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 eventtrace 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 prioritycache 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 test 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 cephfs_mirror 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 cephsqlite 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 crimson_interrupt 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_onode 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_odata 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_omap 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_tm 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_t 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_cleaner 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_epm 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_lba 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_fixedkv_tree 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_cache 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_journal 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_device 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 seastore_backref 2026-02-24T18:19:34.835 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 alienstore 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 mclock 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: 0/ 5 cyanstore 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 ceph_exporter 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 memstore 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: 1/ 5 trace 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: -2/-2 (syslog threshold) 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: 99/99 (stderr threshold) 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr:--- pthread ID / name mapping for recent threads --- 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: 7f878734fbc0 / ceph-kvstore-to 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: max_recent 500 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: max_new 1000 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr: log_file /var/lib/ceph/crash/2026-02-24T18:19:34.787324Z_39e748fa-81d8-4fa2-875f-803970ae6d13/log 2026-02-24T18:19:34.836 INFO:tasks.workunit.client.0.trial020.stderr:--- end dump of recent events --- 2026-02-24T18:19:34.856 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh: line 27: 21915 Segmentation fault (core dumped) ceph-kvstore-tool bluestore-kv ${TEMP_DIR} destructive-repair 2026-02-24T18:19:34.856 INFO:tasks.workunit.client.0.trial020.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:1: test_ceph_kvstore_tool: rm -fr ./cephtool.ZWT 2026-02-24T18:19:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 139 2026-02-24T18:19:34.858 INFO:tasks.workunit:Stopping ['cephtool/test_kvstore_tool.sh'] on client.0... 2026-02-24T18:19:34.858 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-24T18:19:35.167 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/workunit.py", line 144, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/workunit.py", line 292, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/workunit.py", line 433, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial020 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' 2026-02-24T18:19:35.171 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-24T18:19:35.233 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T18:19:35.233 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:19:35.385 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:19:35.386 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T18:19:35.396 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-24T18:19:34.228755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110122+0000","last_change":"2026-02-24T18:18:52.253698+0000","last_active":"2026-02-24T18:18:59.110122+0000","last_peered":"2026-02-24T18:18:59.110122+0000","last_clean":"2026-02-24T18:18:59.110122+0000","last_became_active":"2026-02-24T18:18:50.249066+0000","last_became_peered":"2026-02-24T18:18:50.249066+0000","last_unstale":"2026-02-24T18:18:59.110122+0000","last_undegraded":"2026-02-24T18:18:59.110122+0000","last_fullsized":"2026-02-24T18:18:59.110122+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:46:05.765794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070762199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110119+0000","last_change":"2026-02-24T18:18:52.253607+0000","last_active":"2026-02-24T18:18:59.110119+0000","last_peered":"2026-02-24T18:18:59.110119+0000","last_clean":"2026-02-24T18:18:59.110119+0000","last_became_active":"2026-02-24T18:18:50.249405+0000","last_became_peered":"2026-02-24T18:18:50.249405+0000","last_unstale":"2026-02-24T18:18:59.110119+0000","last_undegraded":"2026-02-24T18:18:59.110119+0000","last_fullsized":"2026-02-24T18:18:59.110119+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:36:13.285373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064563700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569852+0000","last_change":"2026-02-24T18:18:52.250241+0000","last_active":"2026-02-24T18:18:59.569852+0000","last_peered":"2026-02-24T18:18:59.569852+0000","last_clean":"2026-02-24T18:18:59.569852+0000","last_became_active":"2026-02-24T18:18:50.249055+0000","last_became_peered":"2026-02-24T18:18:50.249055+0000","last_unstale":"2026-02-24T18:18:59.569852+0000","last_undegraded":"2026-02-24T18:18:59.569852+0000","last_fullsized":"2026-02-24T18:18:59.569852+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:21:31.791793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033314800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253562+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249327+0000","last_became_peered":"2026-02-24T18:18:50.249327+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:37:29.642093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065132700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253485+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249028+0000","last_became_peered":"2026-02-24T18:18:50.249028+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:49:57.350554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057687899999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.266008+0000","last_change":"2026-02-24T18:18:52.252032+0000","last_active":"2026-02-24T18:18:59.266008+0000","last_peered":"2026-02-24T18:18:59.266008+0000","last_clean":"2026-02-24T18:18:59.266008+0000","last_became_active":"2026-02-24T18:18:50.248784+0000","last_became_peered":"2026-02-24T18:18:50.248784+0000","last_unstale":"2026-02-24T18:18:59.266008+0000","last_undegraded":"2026-02-24T18:18:59.266008+0000","last_fullsized":"2026-02-24T18:18:59.266008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:02:07.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064190000000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569864+0000","last_change":"2026-02-24T18:18:52.250180+0000","last_active":"2026-02-24T18:18:59.569864+0000","last_peered":"2026-02-24T18:18:59.569864+0000","last_clean":"2026-02-24T18:18:59.569864+0000","last_became_active":"2026-02-24T18:18:50.249075+0000","last_became_peered":"2026-02-24T18:18:50.249075+0000","last_unstale":"2026-02-24T18:18:59.569864+0000","last_undegraded":"2026-02-24T18:18:59.569864+0000","last_fullsized":"2026-02-24T18:18:59.569864+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:02:59.019103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028864500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110095+0000","last_change":"2026-02-24T18:18:52.253414+0000","last_active":"2026-02-24T18:18:59.110095+0000","last_peered":"2026-02-24T18:18:59.110095+0000","last_clean":"2026-02-24T18:18:59.110095+0000","last_became_active":"2026-02-24T18:18:50.249535+0000","last_became_peered":"2026-02-24T18:18:50.249535+0000","last_unstale":"2026-02-24T18:18:59.110095+0000","last_undegraded":"2026-02-24T18:18:59.110095+0000","last_fullsized":"2026-02-24T18:18:59.110095+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:08:53.720990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055293499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:19:35.397 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:19:35.542 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:19:35.543 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T18:19:35.555 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-24T18:19:34.228755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110122+0000","last_change":"2026-02-24T18:18:52.253698+0000","last_active":"2026-02-24T18:18:59.110122+0000","last_peered":"2026-02-24T18:18:59.110122+0000","last_clean":"2026-02-24T18:18:59.110122+0000","last_became_active":"2026-02-24T18:18:50.249066+0000","last_became_peered":"2026-02-24T18:18:50.249066+0000","last_unstale":"2026-02-24T18:18:59.110122+0000","last_undegraded":"2026-02-24T18:18:59.110122+0000","last_fullsized":"2026-02-24T18:18:59.110122+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:46:05.765794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070762199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110119+0000","last_change":"2026-02-24T18:18:52.253607+0000","last_active":"2026-02-24T18:18:59.110119+0000","last_peered":"2026-02-24T18:18:59.110119+0000","last_clean":"2026-02-24T18:18:59.110119+0000","last_became_active":"2026-02-24T18:18:50.249405+0000","last_became_peered":"2026-02-24T18:18:50.249405+0000","last_unstale":"2026-02-24T18:18:59.110119+0000","last_undegraded":"2026-02-24T18:18:59.110119+0000","last_fullsized":"2026-02-24T18:18:59.110119+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:36:13.285373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064563700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569852+0000","last_change":"2026-02-24T18:18:52.250241+0000","last_active":"2026-02-24T18:18:59.569852+0000","last_peered":"2026-02-24T18:18:59.569852+0000","last_clean":"2026-02-24T18:18:59.569852+0000","last_became_active":"2026-02-24T18:18:50.249055+0000","last_became_peered":"2026-02-24T18:18:50.249055+0000","last_unstale":"2026-02-24T18:18:59.569852+0000","last_undegraded":"2026-02-24T18:18:59.569852+0000","last_fullsized":"2026-02-24T18:18:59.569852+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:21:31.791793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033314800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253562+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249327+0000","last_became_peered":"2026-02-24T18:18:50.249327+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:37:29.642093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065132700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253485+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249028+0000","last_became_peered":"2026-02-24T18:18:50.249028+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:49:57.350554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057687899999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.266008+0000","last_change":"2026-02-24T18:18:52.252032+0000","last_active":"2026-02-24T18:18:59.266008+0000","last_peered":"2026-02-24T18:18:59.266008+0000","last_clean":"2026-02-24T18:18:59.266008+0000","last_became_active":"2026-02-24T18:18:50.248784+0000","last_became_peered":"2026-02-24T18:18:50.248784+0000","last_unstale":"2026-02-24T18:18:59.266008+0000","last_undegraded":"2026-02-24T18:18:59.266008+0000","last_fullsized":"2026-02-24T18:18:59.266008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:02:07.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064190000000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569864+0000","last_change":"2026-02-24T18:18:52.250180+0000","last_active":"2026-02-24T18:18:59.569864+0000","last_peered":"2026-02-24T18:18:59.569864+0000","last_clean":"2026-02-24T18:18:59.569864+0000","last_became_active":"2026-02-24T18:18:50.249075+0000","last_became_peered":"2026-02-24T18:18:50.249075+0000","last_unstale":"2026-02-24T18:18:59.569864+0000","last_undegraded":"2026-02-24T18:18:59.569864+0000","last_fullsized":"2026-02-24T18:18:59.569864+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:02:59.019103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028864500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110095+0000","last_change":"2026-02-24T18:18:52.253414+0000","last_active":"2026-02-24T18:18:59.110095+0000","last_peered":"2026-02-24T18:18:59.110095+0000","last_clean":"2026-02-24T18:18:59.110095+0000","last_became_active":"2026-02-24T18:18:50.249535+0000","last_became_peered":"2026-02-24T18:18:50.249535+0000","last_unstale":"2026-02-24T18:18:59.110095+0000","last_undegraded":"2026-02-24T18:18:59.110095+0000","last_fullsized":"2026-02-24T18:18:59.110095+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:08:53.720990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055293499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:19:35.556 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T18:19:35.556 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:19:35.699 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:19:35.699 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T18:19:35.709 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-24T18:19:34.228755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001663"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110122+0000","last_change":"2026-02-24T18:18:52.253698+0000","last_active":"2026-02-24T18:18:59.110122+0000","last_peered":"2026-02-24T18:18:59.110122+0000","last_clean":"2026-02-24T18:18:59.110122+0000","last_became_active":"2026-02-24T18:18:50.249066+0000","last_became_peered":"2026-02-24T18:18:50.249066+0000","last_unstale":"2026-02-24T18:18:59.110122+0000","last_undegraded":"2026-02-24T18:18:59.110122+0000","last_fullsized":"2026-02-24T18:18:59.110122+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:46:05.765794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070762199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110119+0000","last_change":"2026-02-24T18:18:52.253607+0000","last_active":"2026-02-24T18:18:59.110119+0000","last_peered":"2026-02-24T18:18:59.110119+0000","last_clean":"2026-02-24T18:18:59.110119+0000","last_became_active":"2026-02-24T18:18:50.249405+0000","last_became_peered":"2026-02-24T18:18:50.249405+0000","last_unstale":"2026-02-24T18:18:59.110119+0000","last_undegraded":"2026-02-24T18:18:59.110119+0000","last_fullsized":"2026-02-24T18:18:59.110119+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:36:13.285373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064563700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569852+0000","last_change":"2026-02-24T18:18:52.250241+0000","last_active":"2026-02-24T18:18:59.569852+0000","last_peered":"2026-02-24T18:18:59.569852+0000","last_clean":"2026-02-24T18:18:59.569852+0000","last_became_active":"2026-02-24T18:18:50.249055+0000","last_became_peered":"2026-02-24T18:18:50.249055+0000","last_unstale":"2026-02-24T18:18:59.569852+0000","last_undegraded":"2026-02-24T18:18:59.569852+0000","last_fullsized":"2026-02-24T18:18:59.569852+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:21:31.791793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033314800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253562+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249327+0000","last_became_peered":"2026-02-24T18:18:50.249327+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:37:29.642093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065132700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253485+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249028+0000","last_became_peered":"2026-02-24T18:18:50.249028+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:49:57.350554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057687899999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.266008+0000","last_change":"2026-02-24T18:18:52.252032+0000","last_active":"2026-02-24T18:18:59.266008+0000","last_peered":"2026-02-24T18:18:59.266008+0000","last_clean":"2026-02-24T18:18:59.266008+0000","last_became_active":"2026-02-24T18:18:50.248784+0000","last_became_peered":"2026-02-24T18:18:50.248784+0000","last_unstale":"2026-02-24T18:18:59.266008+0000","last_undegraded":"2026-02-24T18:18:59.266008+0000","last_fullsized":"2026-02-24T18:18:59.266008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:02:07.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064190000000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569864+0000","last_change":"2026-02-24T18:18:52.250180+0000","last_active":"2026-02-24T18:18:59.569864+0000","last_peered":"2026-02-24T18:18:59.569864+0000","last_clean":"2026-02-24T18:18:59.569864+0000","last_became_active":"2026-02-24T18:18:50.249075+0000","last_became_peered":"2026-02-24T18:18:50.249075+0000","last_unstale":"2026-02-24T18:18:59.569864+0000","last_undegraded":"2026-02-24T18:18:59.569864+0000","last_fullsized":"2026-02-24T18:18:59.569864+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:02:59.019103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028864500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110095+0000","last_change":"2026-02-24T18:18:52.253414+0000","last_active":"2026-02-24T18:18:59.110095+0000","last_peered":"2026-02-24T18:18:59.110095+0000","last_clean":"2026-02-24T18:18:59.110095+0000","last_became_active":"2026-02-24T18:18:50.249535+0000","last_became_peered":"2026-02-24T18:18:50.249535+0000","last_unstale":"2026-02-24T18:18:59.110095+0000","last_undegraded":"2026-02-24T18:18:59.110095+0000","last_fullsized":"2026-02-24T18:18:59.110095+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:08:53.720990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055293499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:19:35.711 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T18:19:35.851 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:19:35.851 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":13,"fsid":"135e2b35-ac06-4607-a4ba-6aa45d264e7b","created":"2026-02-24T18:18:34.147883+0000","modified":"2026-02-24T18:18:59.108344+0000","last_up_change":"2026-02-24T18:18:42.227208+0000","last_in_change":"2026-02-24T18:18:34.744762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-24T18:18:48.384496+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41405e39-39b2-4123-b663-1283aad326f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6818","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6819","nonce":1817351360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6820","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6821","nonce":1817351360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6824","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6825","nonce":1817351360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6822","nonce":1817351360},{"type":"v1","addr":"10.20.193.20:6823","nonce":1817351360}]},"public_addr":"10.20.193.20:6819/1817351360","cluster_addr":"10.20.193.20:6821/1817351360","heartbeat_back_addr":"10.20.193.20:6825/1817351360","heartbeat_front_addr":"10.20.193.20:6823/1817351360","state":["exists","up"]},{"osd":1,"uuid":"4827b0fa-0f0a-4731-b219-2bafdbf7bac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6802","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6803","nonce":1077990978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6804","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6805","nonce":1077990978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6808","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6809","nonce":1077990978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6806","nonce":1077990978},{"type":"v1","addr":"10.20.193.20:6807","nonce":1077990978}]},"public_addr":"10.20.193.20:6803/1077990978","cluster_addr":"10.20.193.20:6805/1077990978","heartbeat_back_addr":"10.20.193.20:6809/1077990978","heartbeat_front_addr":"10.20.193.20:6807/1077990978","state":["exists","up"]},{"osd":2,"uuid":"2fbc4bb6-353f-4de0-a383-c0c6c207f765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6810","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6811","nonce":2532477871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6812","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6813","nonce":2532477871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6816","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6817","nonce":2532477871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6814","nonce":2532477871},{"type":"v1","addr":"10.20.193.20:6815","nonce":2532477871}]},"public_addr":"10.20.193.20:6811/2532477871","cluster_addr":"10.20.193.20:6813/2532477871","heartbeat_back_addr":"10.20.193.20:6817/2532477871","heartbeat_front_addr":"10.20.193.20:6815/2532477871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T18:19:36.862 INFO:tasks.ceph:Scrubbing osd.0 2026-02-24T18:19:36.863 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:19:36.934 INFO:teuthology.orchestra.run.trial020.stdout:{ 2026-02-24T18:19:36.934 INFO:teuthology.orchestra.run.trial020.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:19:36.935 INFO:teuthology.orchestra.run.trial020.stdout:} 2026-02-24T18:19:36.944 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-24T18:19:37.093 INFO:teuthology.orchestra.run.trial020.stderr:instructed osd(s) 0 to deep-scrub 2026-02-24T18:19:37.106 INFO:tasks.ceph:Scrubbing osd.1 2026-02-24T18:19:37.106 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:19:37.185 INFO:teuthology.orchestra.run.trial020.stdout:{ 2026-02-24T18:19:37.185 INFO:teuthology.orchestra.run.trial020.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:19:37.185 INFO:teuthology.orchestra.run.trial020.stdout:} 2026-02-24T18:19:37.194 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-24T18:19:37.347 INFO:teuthology.orchestra.run.trial020.stderr:instructed osd(s) 1 to deep-scrub 2026-02-24T18:19:37.358 INFO:tasks.ceph:Scrubbing osd.2 2026-02-24T18:19:37.358 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-24T18:19:37.434 INFO:teuthology.orchestra.run.trial020.stdout:{ 2026-02-24T18:19:37.434 INFO:teuthology.orchestra.run.trial020.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-24T18:19:37.434 INFO:teuthology.orchestra.run.trial020.stdout:} 2026-02-24T18:19:37.443 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-24T18:19:37.591 INFO:teuthology.orchestra.run.trial020.stderr:instructed osd(s) 2 to deep-scrub 2026-02-24T18:19:37.603 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:19:37.746 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:19:37.747 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T18:19:37.756 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2026-02-24T18:19:36.229015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81236,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491564,"statfs":{"total":322122547200,"available":322039361536,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001711"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110122+0000","last_change":"2026-02-24T18:18:52.253698+0000","last_active":"2026-02-24T18:18:59.110122+0000","last_peered":"2026-02-24T18:18:59.110122+0000","last_clean":"2026-02-24T18:18:59.110122+0000","last_became_active":"2026-02-24T18:18:50.249066+0000","last_became_peered":"2026-02-24T18:18:50.249066+0000","last_unstale":"2026-02-24T18:18:59.110122+0000","last_undegraded":"2026-02-24T18:18:59.110122+0000","last_fullsized":"2026-02-24T18:18:59.110122+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:46:05.765794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070762199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110119+0000","last_change":"2026-02-24T18:18:52.253607+0000","last_active":"2026-02-24T18:18:59.110119+0000","last_peered":"2026-02-24T18:18:59.110119+0000","last_clean":"2026-02-24T18:18:59.110119+0000","last_became_active":"2026-02-24T18:18:50.249405+0000","last_became_peered":"2026-02-24T18:18:50.249405+0000","last_unstale":"2026-02-24T18:18:59.110119+0000","last_undegraded":"2026-02-24T18:18:59.110119+0000","last_fullsized":"2026-02-24T18:18:59.110119+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T02:36:13.285373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064563700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569852+0000","last_change":"2026-02-24T18:18:52.250241+0000","last_active":"2026-02-24T18:18:59.569852+0000","last_peered":"2026-02-24T18:18:59.569852+0000","last_clean":"2026-02-24T18:18:59.569852+0000","last_became_active":"2026-02-24T18:18:50.249055+0000","last_became_peered":"2026-02-24T18:18:50.249055+0000","last_unstale":"2026-02-24T18:18:59.569852+0000","last_undegraded":"2026-02-24T18:18:59.569852+0000","last_fullsized":"2026-02-24T18:18:59.569852+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:21:31.791793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033314800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253562+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249327+0000","last_became_peered":"2026-02-24T18:18:50.249327+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:37:29.642093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065132700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":22,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110104+0000","last_change":"2026-02-24T18:18:52.253485+0000","last_active":"2026-02-24T18:18:59.110104+0000","last_peered":"2026-02-24T18:18:59.110104+0000","last_clean":"2026-02-24T18:18:59.110104+0000","last_became_active":"2026-02-24T18:18:50.249028+0000","last_became_peered":"2026-02-24T18:18:50.249028+0000","last_unstale":"2026-02-24T18:18:59.110104+0000","last_undegraded":"2026-02-24T18:18:59.110104+0000","last_fullsized":"2026-02-24T18:18:59.110104+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:49:57.350554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057687899999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.266008+0000","last_change":"2026-02-24T18:18:52.252032+0000","last_active":"2026-02-24T18:18:59.266008+0000","last_peered":"2026-02-24T18:18:59.266008+0000","last_clean":"2026-02-24T18:18:59.266008+0000","last_became_active":"2026-02-24T18:18:50.248784+0000","last_became_peered":"2026-02-24T18:18:50.248784+0000","last_unstale":"2026-02-24T18:18:59.266008+0000","last_undegraded":"2026-02-24T18:18:59.266008+0000","last_fullsized":"2026-02-24T18:18:59.266008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T03:02:07.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064190000000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.569864+0000","last_change":"2026-02-24T18:18:52.250180+0000","last_active":"2026-02-24T18:18:59.569864+0000","last_peered":"2026-02-24T18:18:59.569864+0000","last_clean":"2026-02-24T18:18:59.569864+0000","last_became_active":"2026-02-24T18:18:50.249075+0000","last_became_peered":"2026-02-24T18:18:50.249075+0000","last_unstale":"2026-02-24T18:18:59.569864+0000","last_undegraded":"2026-02-24T18:18:59.569864+0000","last_fullsized":"2026-02-24T18:18:59.569864+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:02:59.019103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028864500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:18:59.110095+0000","last_change":"2026-02-24T18:18:52.253414+0000","last_active":"2026-02-24T18:18:59.110095+0000","last_peered":"2026-02-24T18:18:59.110095+0000","last_clean":"2026-02-24T18:18:59.110095+0000","last_became_active":"2026-02-24T18:18:50.249535+0000","last_became_peered":"2026-02-24T18:18:50.249535+0000","last_unstale":"2026-02-24T18:18:59.110095+0000","last_undegraded":"2026-02-24T18:18:59.110095+0000","last_fullsized":"2026-02-24T18:18:59.110095+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:18:49.239489+0000","last_clean_scrub_stamp":"2026-02-24T18:18:49.239489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T20:08:53.720990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055293499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27080,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830520,"statfs":{"total":107374182400,"available":107346452480,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:19:37.761 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.762 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.762 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.762 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.762 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.762 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.762 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.763 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-24T18:18:49.239489+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=18, tm_sec=49, tm_wday=1, tm_yday=55, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=24, tm_hour=18, tm_min=19, tm_sec=35, tm_wday=1, tm_yday=55, tm_isdst=0) 2026-02-24T18:19:37.763 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-24T18:19:57.763 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T18:19:57.914 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T18:19:57.914 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T18:19:57.924 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2026-02-24T18:19:57.187255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81224,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":80827,"kb_avail":314491576,"statfs":{"total":322122547200,"available":322039373824,"internally_reserved":0,"allocated":352256,"data_stored":90044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82767187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.956895"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:38.064475+0000","last_change":"2026-02-24T18:19:38.064490+0000","last_active":"2026-02-24T18:19:38.064475+0000","last_peered":"2026-02-24T18:19:38.064475+0000","last_clean":"2026-02-24T18:19:38.064475+0000","last_became_active":"2026-02-24T18:18:50.249066+0000","last_became_peered":"2026-02-24T18:18:50.249066+0000","last_unstale":"2026-02-24T18:19:38.064475+0000","last_undegraded":"2026-02-24T18:19:38.064475+0000","last_fullsized":"2026-02-24T18:19:38.064475+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:19:38.064423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:19:38.064423+0000","last_clean_scrub_stamp":"2026-02-24T18:19:38.064423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:04:55.902331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070762199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:39.040151+0000","last_change":"2026-02-24T18:19:39.040166+0000","last_active":"2026-02-24T18:19:39.040151+0000","last_peered":"2026-02-24T18:19:39.040151+0000","last_clean":"2026-02-24T18:19:39.040151+0000","last_became_active":"2026-02-24T18:18:50.249405+0000","last_became_peered":"2026-02-24T18:18:50.249405+0000","last_unstale":"2026-02-24T18:19:39.040151+0000","last_undegraded":"2026-02-24T18:19:39.040151+0000","last_fullsized":"2026-02-24T18:19:39.040151+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:19:39.040098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:19:39.040098+0000","last_clean_scrub_stamp":"2026-02-24T18:19:39.040098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T05:40:53.866891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064563700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:38.367666+0000","last_change":"2026-02-24T18:19:38.367687+0000","last_active":"2026-02-24T18:19:38.367666+0000","last_peered":"2026-02-24T18:19:38.367666+0000","last_clean":"2026-02-24T18:19:38.367666+0000","last_became_active":"2026-02-24T18:18:50.249055+0000","last_became_peered":"2026-02-24T18:18:50.249055+0000","last_unstale":"2026-02-24T18:19:38.367666+0000","last_undegraded":"2026-02-24T18:19:38.367666+0000","last_fullsized":"2026-02-24T18:19:38.367666+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:19:38.367615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:19:38.367615+0000","last_clean_scrub_stamp":"2026-02-24T18:19:38.367615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T19:09:32.448218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033314800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:39.990183+0000","last_change":"2026-02-24T18:19:39.990200+0000","last_active":"2026-02-24T18:19:39.990183+0000","last_peered":"2026-02-24T18:19:39.990183+0000","last_clean":"2026-02-24T18:19:39.990183+0000","last_became_active":"2026-02-24T18:18:50.249327+0000","last_became_peered":"2026-02-24T18:18:50.249327+0000","last_unstale":"2026-02-24T18:19:39.990183+0000","last_undegraded":"2026-02-24T18:19:39.990183+0000","last_fullsized":"2026-02-24T18:19:39.990183+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:19:39.990135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:19:39.990135+0000","last_clean_scrub_stamp":"2026-02-24T18:19:39.990135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-25T22:11:35.684427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065132700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"13'2","reported_seq":32,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:40.961477+0000","last_change":"2026-02-24T18:19:40.961496+0000","last_active":"2026-02-24T18:19:40.961477+0000","last_peered":"2026-02-24T18:19:40.961477+0000","last_clean":"2026-02-24T18:19:40.961477+0000","last_became_active":"2026-02-24T18:18:50.249028+0000","last_became_peered":"2026-02-24T18:18:50.249028+0000","last_unstale":"2026-02-24T18:19:40.961477+0000","last_undegraded":"2026-02-24T18:19:40.961477+0000","last_fullsized":"2026-02-24T18:19:40.961477+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'2","last_scrub_stamp":"2026-02-24T18:19:40.961435+0000","last_deep_scrub":"13'2","last_deep_scrub_stamp":"2026-02-24T18:19:40.961435+0000","last_clean_scrub_stamp":"2026-02-24T18:19:40.961435+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T04:23:26.250138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057687899999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'3","reported_seq":33,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:37.169610+0000","last_change":"2026-02-24T18:19:37.169628+0000","last_active":"2026-02-24T18:19:37.169610+0000","last_peered":"2026-02-24T18:19:37.169610+0000","last_clean":"2026-02-24T18:19:37.169610+0000","last_became_active":"2026-02-24T18:18:50.248784+0000","last_became_peered":"2026-02-24T18:18:50.248784+0000","last_unstale":"2026-02-24T18:19:37.169610+0000","last_undegraded":"2026-02-24T18:19:37.169610+0000","last_fullsized":"2026-02-24T18:19:37.169610+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'3","last_scrub_stamp":"2026-02-24T18:19:37.169575+0000","last_deep_scrub":"13'3","last_deep_scrub_stamp":"2026-02-24T18:19:37.169575+0000","last_clean_scrub_stamp":"2026-02-24T18:19:37.169575+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T00:14:00.506756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064190000000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":28,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:39.320978+0000","last_change":"2026-02-24T18:19:39.320996+0000","last_active":"2026-02-24T18:19:39.320978+0000","last_peered":"2026-02-24T18:19:39.320978+0000","last_clean":"2026-02-24T18:19:39.320978+0000","last_became_active":"2026-02-24T18:18:50.249075+0000","last_became_peered":"2026-02-24T18:18:50.249075+0000","last_unstale":"2026-02-24T18:19:39.320978+0000","last_undegraded":"2026-02-24T18:19:39.320978+0000","last_fullsized":"2026-02-24T18:19:39.320978+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:19:39.320923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:19:39.320923+0000","last_clean_scrub_stamp":"2026-02-24T18:19:39.320923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T01:12:06.445345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028864500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":30,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-24T18:19:41.998739+0000","last_change":"2026-02-24T18:19:41.998754+0000","last_active":"2026-02-24T18:19:41.998739+0000","last_peered":"2026-02-24T18:19:41.998739+0000","last_clean":"2026-02-24T18:19:41.998739+0000","last_became_active":"2026-02-24T18:18:50.249535+0000","last_became_peered":"2026-02-24T18:18:50.249535+0000","last_unstale":"2026-02-24T18:19:41.998739+0000","last_undegraded":"2026-02-24T18:19:41.998739+0000","last_fullsized":"2026-02-24T18:19:41.998739+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T18:19:41.998689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T18:19:41.998689+0000","last_clean_scrub_stamp":"2026-02-24T18:19:41.998689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-26T01:45:57.991146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055293499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738385,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27072,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830528,"statfs":{"total":107374182400,"available":107346460672,"internally_reserved":0,"allocated":114688,"data_stored":30002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738385,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738386,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27076,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":104830524,"statfs":{"total":107374182400,"available":107346456576,"internally_reserved":0,"allocated":118784,"data_stored":30021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T18:19:57.926 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-24T18:19:58.088 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-24T18:19:58.088 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-24T18:19:58.089 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-24T18:19:58.089 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:19:58.162 INFO:tasks.ceph.osd.0:Stopped 2026-02-24T18:19:58.163 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-24T18:19:58.163 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:19:58.247 INFO:tasks.ceph.osd.1:Stopped 2026-02-24T18:19:58.247 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-24T18:19:58.248 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:19:58.342 INFO:tasks.ceph.osd.2:Stopped 2026-02-24T18:19:58.343 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-24T18:19:58.343 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-24T18:19:58.343 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:19:58.369 INFO:tasks.ceph.mgr.x:Stopped 2026-02-24T18:19:58.369 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-24T18:19:58.369 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-24T18:19:58.369 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T18:19:58.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-24T18:19:58.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2026-02-24T18:19:58.424 INFO:tasks.ceph.mon.a:Stopped 2026-02-24T18:19:58.424 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-24T18:19:58.425 DEBUG:teuthology.orchestra.run.trial020:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-24T18:19:58.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:19:58.477 DEBUG:teuthology.orchestra.run.trial020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-24T18:19:58.531 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:19:58.532 DEBUG:teuthology.orchestra.run.trial020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-24T18:19:58.586 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:19:58.586 DEBUG:teuthology.orchestra.run.trial020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-24T18:19:58.641 INFO:tasks.ceph:Archiving mon data... 2026-02-24T18:19:58.643 DEBUG:teuthology.misc:Transferring archived files from trial020:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642/data/mon.a.tgz 2026-02-24T18:19:58.643 DEBUG:teuthology.orchestra.run.trial020:> mktemp 2026-02-24T18:19:58.686 INFO:teuthology.orchestra.run.trial020.stdout:/tmp/tmp.YVOModsPRu 2026-02-24T18:19:58.687 DEBUG:teuthology.orchestra.run.trial020:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YVOModsPRu 2026-02-24T18:19:58.749 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0666 /tmp/tmp.YVOModsPRu 2026-02-24T18:19:58.844 DEBUG:teuthology.orchestra.remote:trial020:/tmp/tmp.YVOModsPRu is 129KB 2026-02-24T18:19:58.856 DEBUG:teuthology.orchestra.run.trial020:> rm -fr /tmp/tmp.YVOModsPRu 2026-02-24T18:19:58.860 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-24T18:19:58.861 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-24T18:19:58.955 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-24T18:19:58.955 INFO:tasks.ceph:Archiving crash dumps... 2026-02-24T18:19:58.957 DEBUG:teuthology.misc:Transferring archived files from trial020:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642/remote/trial020/crash 2026-02-24T18:19:58.957 DEBUG:teuthology.orchestra.run.trial020:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-24T18:19:59.003 INFO:tasks.ceph:Compressing logs... 2026-02-24T18:19:59.003 DEBUG:teuthology.orchestra.run.trial020:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T18:19:59.055 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16248.log 2026-02-24T18:19:59.057 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22143.log 2026-02-24T18:19:59.057 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16606.log 2026-02-24T18:19:59.057 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16248.log.gz 2026-02-24T18:19:59.057 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22221.log 2026-02-24T18:19:59.057 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22143.log.gz 2026-02-24T18:19:59.057 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16606.log.gz 2026-02-24T18:19:59.057 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22351.log 2026-02-24T18:19:59.058 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14620.log 2026-02-24T18:19:59.058 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22221.log: /var/log/ceph/ceph-client.admin.22351.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22351.log.gz 2026-02-24T18:19:59.058 INFO:teuthology.orchestra.run.trial020.stderr: -- replaced with /var/log/ceph/ceph-client.admin.22221.log.gz 2026-02-24T18:19:59.058 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-24T18:19:59.064 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.14620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16405.log 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.14620.log.gz 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.16144.log 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16405.log.gz 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16070.log 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16144.log.gz 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16216.log 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22325.log 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16070.log.gz 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16216.log.gz 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14471.log 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22377.log 2026-02-24T18:19:59.065 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22325.log.gz 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.14471.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.14471.log.gz 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22377.log.gz 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14519.log 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14495.log 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.14519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14417.log 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.14519.log.gz 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.14495.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.14495.log.gz 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.16094.log 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.14417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16119.log 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.14417.log.gz 2026-02-24T18:19:59.066 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16328.log 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16094.log.gz 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16250.log 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16119.log.gz 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16328.log.gz 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16327.log 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16250.log.gz 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16327.log.gz 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14578.log 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr: 87.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.14578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.14578.log.gz 2026-02-24T18:19:59.067 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16190.log 2026-02-24T18:19:59.068 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16506.log 2026-02-24T18:19:59.068 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16164.log 2026-02-24T18:19:59.068 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16190.log.gz 2026-02-24T18:19:59.068 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16506.log.gz 2026-02-24T18:19:59.068 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22195.log 2026-02-24T18:19:59.068 INFO:teuthology.orchestra.run.trial020.stderr:gzip/var/log/ceph/ceph-client.admin.16164.log: -5 --verbose -- /var/log/ceph/ceph.log 2026-02-24T18:19:59.068 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16164.log.gz 2026-02-24T18:19:59.073 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22453.log 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22195.log.gz 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16554.log 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22453.log: 83.2% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22453.log.gz 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16249.log 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22169.log 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16554.log: /var/log/ceph/ceph-client.admin.16249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16554.log.gz 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16249.log.gz 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.11870.log 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22429.log 2026-02-24T18:19:59.074 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22169.log.gz 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph.tmp-client.admin.11870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22273.log 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.11870.log.gz 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16398.log 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22429.log: /var/log/ceph/ceph-client.admin.22273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22429.log.gz 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22273.log.gz 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22403.log 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16326.log 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16398.log.gz 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.22403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22403.log.gz 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16406.log 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16326.log.gz 2026-02-24T18:19:59.075 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16476.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16406.log.gz 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.14351.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22247.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16476.log.gz 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16480.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.14351.log: 0.0%/var/log/ceph/ceph-client.admin.22247.log: -- replaced with /var/log/ceph/ceph-client.admin.14351.log.gz 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22247.log.gz 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16480.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.16480.log.gz /var/log/ceph/ceph-client.admin.16580.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22299.log 2026-02-24T18:19:59.076 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.16580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16580.log.gz 2026-02-24T18:19:59.077 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.22299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22299.log.gz 2026-02-24T18:19:59.077 INFO:teuthology.orchestra.run.trial020.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-24T18:19:59.083 INFO:teuthology.orchestra.run.trial020.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-24T18:19:59.083 INFO:teuthology.orchestra.run.trial020.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-24T18:19:59.092 INFO:teuthology.orchestra.run.trial020.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-24T18:19:59.092 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T18:19:59.093 INFO:teuthology.orchestra.run.trial020.stderr:real 0m0.044s 2026-02-24T18:19:59.093 INFO:teuthology.orchestra.run.trial020.stderr:user 0m0.107s 2026-02-24T18:19:59.093 INFO:teuthology.orchestra.run.trial020.stderr:sys 0m0.020s 2026-02-24T18:19:59.094 INFO:tasks.ceph:Archiving logs... 2026-02-24T18:19:59.094 DEBUG:teuthology.misc:Transferring archived files from trial020:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642/remote/trial020/log 2026-02-24T18:19:59.095 DEBUG:teuthology.orchestra.run.trial020:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T18:19:59.200 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T18:19:59.220 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-24T18:19:59.221 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T18:19:59.326 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-24T18:19:59.346 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-24T18:19:59.346 DEBUG:teuthology.orchestra.run.trial020:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T18:19:59.413 INFO:teuthology.orchestra.run.trial020.stdout: remote refid st t when poll reach delay offset jitter 2026-02-24T18:19:59.414 INFO:teuthology.orchestra.run.trial020.stdout:============================================================================== 2026-02-24T18:19:59.414 INFO:teuthology.orchestra.run.trial020.stdout: ntp6.kernfusion 237.17.204.95 2 u 61 64 3 3.671 -7.854 0.180 2026-02-24T18:19:59.414 INFO:teuthology.orchestra.run.trial020.stdout: frumentum-zeus. 209.180.247.49 2 u 61 64 3 29.826 -6.256 0.001 2026-02-24T18:19:59.414 INFO:teuthology.orchestra.run.trial020.stdout: time.cloudflare 10.237.8.4 3 u 63 64 3 3.150 -7.800 0.030 2026-02-24T18:19:59.414 INFO:teuthology.orchestra.run.trial020.stdout: ntp.shastacoe.n 66.244.0.68 2 u 59 64 3 76.590 -7.323 0.107 2026-02-24T18:19:59.414 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-24T18:19:59.441 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-24T18:19:59.441 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-24T18:19:59.459 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-24T18:19:59.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-24T18:19:59.493 INFO:teuthology.task.internal:Duration was 283.862832 seconds 2026-02-24T18:19:59.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-24T18:19:59.509 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-24T18:19:59.510 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T18:19:59.538 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-24T18:19:59.538 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial020.front.sepia.ceph.com 2026-02-24T18:19:59.539 DEBUG:teuthology.orchestra.run.trial020:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T18:19:59.589 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-24T18:19:59.589 DEBUG:teuthology.orchestra.run.trial020:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T18:19:59.860 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-24T18:19:59.860 DEBUG:teuthology.orchestra.run.trial020:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T18:19:59.869 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T18:19:59.871 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T18:19:59.871 INFO:teuthology.orchestra.run.trial020.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T18:19:59.871 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T18:19:59.871 INFO:teuthology.orchestra.run.trial020.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T18:19:59.941 INFO:teuthology.orchestra.run.trial020.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T18:19:59.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-24T18:19:59.959 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-24T18:19:59.959 DEBUG:teuthology.orchestra.run.trial020:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T18:19:59.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-24T18:20:00.008 DEBUG:teuthology.orchestra.run.trial020:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T18:20:00.042 INFO:teuthology.orchestra.run.trial020.stdout:kernel.core_pattern = core 2026-02-24T18:20:00.054 DEBUG:teuthology.orchestra.run.trial020:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T18:20:00.098 WARNING:teuthology.task.internal:Found coredumps on ubuntu@trial020.front.sepia.ceph.com, flagging run as failed 2026-02-24T18:20:00.098 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-24T18:20:00.114 INFO:teuthology.task.internal:Transferring archived files... 2026-02-24T18:20:00.115 DEBUG:teuthology.misc:Transferring archived files from trial020:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642/remote/trial020 2026-02-24T18:20:00.116 DEBUG:teuthology.orchestra.run.trial020:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T18:20:02.548 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2026-02-24T18:20:02.553 INFO:teuthology.task.internal: core looks like: /home/teuthworker/mnt/teuthology/yuriw-2026-02-24_16:22:43-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/68642/remote/trial020/coredump/1771957174.21915.core.gz: gzip compressed data, was "1771957174.21915.core", last modified: Tue Feb 24 18:20:00 2026, max compression, original size modulo 2^32 129236992 2026-02-24T18:20:02.553 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2026-02-24T18:20:02.715 INFO:teuthology.task.internal: core looks like: /tmp/tmpzeox62f4: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-kvstore-tool bluestore-kv ./cephtool.ZWT destructive-repair', real uid: 1000, effective uid: 1000, real gid: 1002, effective gid: 1002, execfn: '/usr/bin/ceph-kvstore-tool', platform: 'x86_64' 2026-02-24T18:20:02.717 INFO:teuthology.task.internal: dump_program: ceph-kvstore-tool 2026-02-24T18:20:02.717 DEBUG:teuthology.orchestra.run.trial020:> which ceph-kvstore-tool 2026-02-24T18:20:02.721 INFO:teuthology.orchestra.run.trial020.stdout:/usr/bin/ceph-kvstore-tool 2026-02-24T18:20:02.769 DEBUG:teuthology.orchestra.remote:trial020:/usr/bin/ceph-kvstore-tool is 10MB 2026-02-24T18:20:02.880 DEBUG:teuthology.orchestra.run.trial020:> mktemp 2026-02-24T18:20:02.884 INFO:teuthology.orchestra.run.trial020.stdout:/tmp/tmp.LcbgBbBFGB 2026-02-24T18:20:02.884 DEBUG:teuthology.orchestra.run.trial020:> sudo cp /usr/bin/ceph-kvstore-tool /tmp/tmp.LcbgBbBFGB 2026-02-24T18:20:02.943 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0666 /tmp/tmp.LcbgBbBFGB 2026-02-24T18:20:03.039 DEBUG:teuthology.orchestra.remote:trial020:/tmp/tmp.LcbgBbBFGB is 10MB 2026-02-24T18:20:03.187 DEBUG:teuthology.orchestra.run.trial020:> rm -fr /tmp/tmp.LcbgBbBFGB 2026-02-24T18:20:03.194 INFO:teuthology.task.internal:Removing archive directory... 2026-02-24T18:20:03.194 DEBUG:teuthology.orchestra.run.trial020:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T18:20:03.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-24T18:20:03.262 INFO:teuthology.task.internal:Not uploading archives. 2026-02-24T18:20:03.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-24T18:20:03.277 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-24T18:20:03.277 DEBUG:teuthology.orchestra.run.trial020:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T18:20:03.294 INFO:teuthology.orchestra.run.trial020.stdout: 1311292 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 24 18:20 /home/ubuntu/cephtest 2026-02-24T18:20:03.295 INFO:teuthology.orchestra.run.trial020.stdout: 1320939 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 24 18:18 /home/ubuntu/cephtest/mnt.0 2026-02-24T18:20:03.295 INFO:teuthology.orchestra.run.trial020.stdout: 1320940 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 24 18:19 /home/ubuntu/cephtest/mnt.0/client.0 2026-02-24T18:20:03.295 INFO:teuthology.orchestra.run.trial020.stdout: 1335689 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 24 18:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-24T18:20:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T18:20:03.296 INFO:teuthology.orchestra.run.trial020.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2026-02-24T18:20:03.297 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial020 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-24T18:20:03.299 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-24T18:20:03.320 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-24T18:20:03.337 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on trial020 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-24T18:20:03.343 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} duration: 283.86283230781555 failure_reason: 'Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial020 with status 139: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh''' flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-24T18:20:03.343 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T18:20:03.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-24T18:20:03.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2026-02-24T18:20:03.471 INFO:teuthology.run:FAIL