2026-02-25T10:08:49.226 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T10:08:49.245 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T10:08:49.350 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115 branch: wip-logstoremanager description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} thrashers/{default} thrashosds-health workloads/write_fadvise_dontneed} email: null first_in_suite: false flavor: debug job_id: '69115' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-logstoremanager ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd target transaction size: 5 seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Degraded data redundancy - is down - osds down - pg .*? is .*?degraded.*?, acting - pg .*? is stuck - pg degraded - PG_BACKFILL_FULL - Low space hindering backfill .*? backfill_toofull - OSD_ROOT_DOWN - pgs degraded - pgs undersized - is active.*backfill_toofull.* sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: c32469671c6b0222f786dd36c34eb702f780a11d owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 869 sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: c32469671c6b0222f786dd36c34eb702f780a11d targets: trial068.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKgWGIuVULmLFMLXdWbNDTQ1G+/PHTTKZPaQZLiqMwp2X6mxIDJlXUENE9nBvitRmZsqzm6pT0Zf0MluMYLKkw= trial072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbarXh6aldkIuv664JPlhZwLEshKDob0pFt7cPRnLfPqgP9wcCTl5wJ7Jrdx9xcz+eA44OmheAwhWNQIgUr7NY= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_trim_stale_osdmaps: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 500 op_weights: write: 100 ops: 4000 write_fadvise_dontneed: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_07:55:35 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T10:08:49.350 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa; will attempt to use it 2026-02-25T10:08:49.350 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks 2026-02-25T10:08:49.350 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T10:08:49.352 INFO:teuthology.task.internal:Checking packages... 2026-02-25T10:08:49.352 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '6aeb6975bc9a4790a399c9cc70dba9635d2a7819' 2026-02-25T10:08:49.352 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T10:08:49.352 INFO:teuthology.packaging:ref: None 2026-02-25T10:08:49.352 INFO:teuthology.packaging:tag: None 2026-02-25T10:08:49.352 INFO:teuthology.packaging:branch: wip-logstoremanager 2026-02-25T10:08:49.352 INFO:teuthology.packaging:sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:08:49.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-logstoremanager 2026-02-25T10:08:49.475 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5338.g6aeb6975 2026-02-25T10:08:49.476 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T10:08:49.487 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T10:08:49.487 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T10:08:49.502 INFO:teuthology.task.internal:Saving configuration 2026-02-25T10:08:49.514 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T10:08:49.526 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T10:08:49.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial068.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 10:05:21.434191', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKgWGIuVULmLFMLXdWbNDTQ1G+/PHTTKZPaQZLiqMwp2X6mxIDJlXUENE9nBvitRmZsqzm6pT0Zf0MluMYLKkw='} 2026-02-25T10:08:49.658 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial072.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 10:05:21.434829', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbarXh6aldkIuv664JPlhZwLEshKDob0pFt7cPRnLfPqgP9wcCTl5wJ7Jrdx9xcz+eA44OmheAwhWNQIgUr7NY='} 2026-02-25T10:08:49.658 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T10:08:49.669 INFO:teuthology.task.internal:roles: ubuntu@trial068.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2026-02-25T10:08:49.669 INFO:teuthology.task.internal:roles: ubuntu@trial072.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-02-25T10:08:49.669 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T10:08:50.046 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x722d14f3d1c0>, signals=[15]) 2026-02-25T10:08:50.046 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T10:08:50.057 INFO:teuthology.task.internal:Opening connections... 2026-02-25T10:08:50.057 DEBUG:teuthology.task.internal:connecting to ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:08:50.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:08:50.121 DEBUG:teuthology.task.internal:connecting to ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:08:50.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:08:50.183 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T10:08:50.211 DEBUG:teuthology.orchestra.run.trial068:> uname -m 2026-02-25T10:08:50.226 INFO:teuthology.orchestra.run.trial068.stdout:x86_64 2026-02-25T10:08:50.226 DEBUG:teuthology.orchestra.run.trial068:> cat /etc/os-release 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:NAME="CentOS Stream" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:VERSION="9" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:ID="centos" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:ID_LIKE="rhel fedora" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:VERSION_ID="9" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:PLATFORM_ID="platform:el9" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:ANSI_COLOR="0;31" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:LOGO="fedora-logo-icon" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:HOME_URL="https://centos.org/" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T10:08:50.281 INFO:teuthology.orchestra.run.trial068.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T10:08:50.282 INFO:teuthology.lock.ops:Updating trial068.front.sepia.ceph.com on lock server 2026-02-25T10:08:50.353 DEBUG:teuthology.orchestra.run.trial072:> uname -m 2026-02-25T10:08:50.368 INFO:teuthology.orchestra.run.trial072.stdout:x86_64 2026-02-25T10:08:50.368 DEBUG:teuthology.orchestra.run.trial072:> cat /etc/os-release 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:NAME="CentOS Stream" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:VERSION="9" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:ID="centos" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:ID_LIKE="rhel fedora" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:VERSION_ID="9" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:PLATFORM_ID="platform:el9" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:ANSI_COLOR="0;31" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:LOGO="fedora-logo-icon" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:HOME_URL="https://centos.org/" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T10:08:50.424 INFO:teuthology.orchestra.run.trial072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T10:08:50.425 INFO:teuthology.lock.ops:Updating trial072.front.sepia.ceph.com on lock server 2026-02-25T10:08:50.561 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T10:08:50.575 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T10:08:50.590 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T10:08:50.590 DEBUG:teuthology.orchestra.run.trial068:> test '!' -e /home/ubuntu/cephtest 2026-02-25T10:08:50.592 DEBUG:teuthology.orchestra.run.trial072:> test '!' -e /home/ubuntu/cephtest 2026-02-25T10:08:50.608 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T10:08:50.619 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T10:08:50.619 DEBUG:teuthology.orchestra.run.trial068:> test -z $(ls -A /var/lib/ceph) 2026-02-25T10:08:50.648 DEBUG:teuthology.orchestra.run.trial072:> test -z $(ls -A /var/lib/ceph) 2026-02-25T10:08:50.666 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T10:08:50.840 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T10:08:50.859 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T10:08:50.859 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T10:08:50.860 DEBUG:teuthology.orchestra.run.trial068:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T10:08:50.860 DEBUG:teuthology.orchestra.run.trial072:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T10:08:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:08:50.875 DEBUG:teuthology.orchestra.run.trial068:> uname -r 2026-02-25T10:08:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:08:50.875 DEBUG:teuthology.orchestra.run.trial072:> uname -r 2026-02-25T10:08:50.930 INFO:teuthology.orchestra.run.trial068.stdout:5.14.0-681.el9.x86_64 2026-02-25T10:08:50.930 INFO:teuthology.task.kernel:Running kernel on trial068: 5.14.0-681.el9.x86_64 2026-02-25T10:08:50.930 DEBUG:teuthology.orchestra.run.trial068:> sudo yum install -y kernel 2026-02-25T10:08:50.931 INFO:teuthology.orchestra.run.trial072.stdout:5.14.0-681.el9.x86_64 2026-02-25T10:08:50.931 INFO:teuthology.task.kernel:Running kernel on trial072: 5.14.0-681.el9.x86_64 2026-02-25T10:08:50.931 DEBUG:teuthology.orchestra.run.trial072:> sudo yum install -y kernel 2026-02-25T10:08:52.024 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-25T10:08:52.686 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.9 MB 00:01 2026-02-25T10:08:56.348 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 27 MB 00:03 2026-02-25T10:09:00.075 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 8.0 MB 00:01 2026-02-25T10:09:01.003 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - Extras packages 107 kB/s | 20 kB 00:00 2026-02-25T10:09:01.439 INFO:teuthology.orchestra.run.trial072.stdout:Extra Packages for Enterprise Linux 54 MB/s | 20 MB 00:00 2026-02-25T10:09:04.592 INFO:teuthology.orchestra.run.trial072.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T10:09:05.595 INFO:teuthology.orchestra.run.trial072.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T10:09:05.595 INFO:teuthology.orchestra.run.trial072.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T10:09:05.615 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout: Package Architecture Version Repository Size 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout:Installing dependencies: 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout:Install 4 Packages 2026-02-25T10:09:05.616 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:09:05.617 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 89 M 2026-02-25T10:09:05.617 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 135 M 2026-02-25T10:09:05.617 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:09:05.841 INFO:teuthology.orchestra.run.trial072.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 5.5 MB/s | 901 kB 00:00 2026-02-25T10:09:05.985 INFO:teuthology.orchestra.run.trial072.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 57 MB/s | 17 MB 00:00 2026-02-25T10:09:06.131 INFO:teuthology.orchestra.run.trial072.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 107 MB/s | 31 MB 00:00 2026-02-25T10:09:06.205 INFO:teuthology.orchestra.run.trial072.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 76 MB/s | 40 MB 00:00 2026-02-25T10:09:06.205 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:09:06.205 INFO:teuthology.orchestra.run.trial072.stdout:Total 151 MB/s | 89 MB 00:00 2026-02-25T10:09:06.394 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:09:06.596 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:09:06.596 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:09:06.980 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:09:06.981 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:09:07.267 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - AppStream 1.8 MB/s | 27 MB 00:14 2026-02-25T10:09:08.092 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:09:08.382 INFO:teuthology.orchestra.run.trial072.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T10:09:08.404 INFO:teuthology.orchestra.run.trial072.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T10:09:10.840 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 8.0 MB 00:01 2026-02-25T10:09:10.978 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T10:09:11.011 INFO:teuthology.orchestra.run.trial072.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T10:09:11.777 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 20 kB 00:00 2026-02-25T10:09:12.198 INFO:teuthology.orchestra.run.trial068.stdout:Extra Packages for Enterprise Linux 57 MB/s | 20 MB 00:00 2026-02-25T10:09:14.113 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T10:09:14.119 INFO:teuthology.orchestra.run.trial072.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:15.383 INFO:teuthology.orchestra.run.trial068.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T10:09:16.391 INFO:teuthology.orchestra.run.trial068.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T10:09:16.392 INFO:teuthology.orchestra.run.trial068.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T10:09:16.411 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T10:09:16.413 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:09:16.414 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:09:16.414 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:09:16.414 INFO:teuthology.orchestra.run.trial068.stdout:Install 4 Packages 2026-02-25T10:09:16.414 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:09:16.414 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 89 M 2026-02-25T10:09:16.414 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 135 M 2026-02-25T10:09:16.414 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:09:17.163 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:18.245 INFO:teuthology.orchestra.run.trial068.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 571 kB/s | 901 kB 00:01 2026-02-25T10:09:22.392 INFO:teuthology.orchestra.run.trial068.stdout:(2/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 7.0 MB/s | 40 MB 00:05 2026-02-25T10:09:22.500 INFO:teuthology.orchestra.run.trial068.stdout:(3/4): kernel-core-5.14.0-686.el9.x86_64.rpm 2.9 MB/s | 17 MB 00:05 2026-02-25T10:09:22.781 INFO:teuthology.orchestra.run.trial068.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 6.8 MB/s | 31 MB 00:04 2026-02-25T10:09:22.783 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:09:22.783 INFO:teuthology.orchestra.run.trial068.stdout:Total 14 MB/s | 89 MB 00:06 2026-02-25T10:09:22.974 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:09:23.178 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:09:23.178 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:09:23.560 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:09:23.561 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:09:24.675 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:09:24.963 INFO:teuthology.orchestra.run.trial068.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T10:09:24.985 INFO:teuthology.orchestra.run.trial068.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T10:09:27.551 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T10:09:27.585 INFO:teuthology.orchestra.run.trial068.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T10:09:30.621 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T10:09:30.627 INFO:teuthology.orchestra.run.trial068.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:33.682 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:34.848 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:34.869 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:35.055 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:35.055 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T10:09:35.055 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T10:09:35.055 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T10:09:35.111 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:35.111 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:09:35.112 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:09:35.112 INFO:teuthology.orchestra.run.trial072.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.112 INFO:teuthology.orchestra.run.trial072.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.112 INFO:teuthology.orchestra.run.trial072.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.112 INFO:teuthology.orchestra.run.trial072.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.112 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:09:35.112 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:09:35.172 DEBUG:teuthology.orchestra.run.trial072:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T10:09:35.200 INFO:teuthology.orchestra.run.trial072.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.200 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T10:09:35.200 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T10:09:35.200 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T10:09:35.200 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T10:09:35.200 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2026-02-25T10:09:35.200 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial072.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T10:09:35.200 DEBUG:teuthology.orchestra.run.trial072:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T10:09:35.266 INFO:teuthology.orchestra.run.trial072.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.267 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T10:09:35.267 DEBUG:teuthology.orchestra.run.trial072:> sudo rpm -qi grub2-tools 2026-02-25T10:09:35.337 INFO:teuthology.orchestra.run.trial072.stdout:Name : grub2-tools 2026-02-25T10:09:35.337 INFO:teuthology.orchestra.run.trial072.stdout:Epoch : 1 2026-02-25T10:09:35.337 INFO:teuthology.orchestra.run.trial072.stdout:Version : 2.06 2026-02-25T10:09:35.337 INFO:teuthology.orchestra.run.trial072.stdout:Release : 120.el9 2026-02-25T10:09:35.337 INFO:teuthology.orchestra.run.trial072.stdout:Architecture: x86_64 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Group : Unspecified 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Size : 9108611 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:License : GPLv3+ 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Packager : builder@centos.org 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Vendor : CentOS 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T10:09:35.338 INFO:teuthology.orchestra.run.trial072.stdout:Summary : Support tools for GRUB. 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout:Description : 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout:hardware devices. 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:09:35.339 INFO:teuthology.orchestra.run.trial072.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T10:09:35.340 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T10:09:35.340 INFO:teuthology.task.kernel:Updating grub on trial072 to boot 5.14.0-686.el9.x86_64 2026-02-25T10:09:35.341 DEBUG:teuthology.orchestra.run.trial072:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T10:09:35.406 INFO:teuthology.orchestra.run.trial072.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T10:09:35.408 DEBUG:teuthology.orchestra.run.trial072:> sudo command -v grubby && echo yes || echo no 2026-02-25T10:09:35.471 INFO:teuthology.orchestra.run.trial072.stdout:/sbin/grubby 2026-02-25T10:09:35.473 INFO:teuthology.orchestra.run.trial072.stdout:yes 2026-02-25T10:09:35.473 DEBUG:teuthology.orchestra.run.trial072:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T10:09:35.537 INFO:teuthology.orchestra.run.trial072.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T10:09:35.538 DEBUG:teuthology.orchestra.run.trial072:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.612 DEBUG:teuthology.orchestra.run.trial072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T10:09:35.823 INFO:teuthology.orchestra.run.trial072.stderr:Generating grub configuration file ... 2026-02-25T10:09:35.890 INFO:teuthology.orchestra.run.trial072.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T10:09:35.897 INFO:teuthology.orchestra.run.trial072.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T10:09:36.150 INFO:teuthology.orchestra.run.trial072.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T10:09:36.157 INFO:teuthology.orchestra.run.trial072.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T10:09:36.190 INFO:teuthology.orchestra.run.trial072.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T10:09:36.197 INFO:teuthology.orchestra.run.trial072.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T10:09:36.229 INFO:teuthology.orchestra.run.trial072.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T10:09:36.236 INFO:teuthology.orchestra.run.trial072.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T10:09:36.259 INFO:teuthology.orchestra.run.trial072.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T10:09:36.265 INFO:teuthology.orchestra.run.trial072.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T10:09:36.353 INFO:teuthology.orchestra.run.trial072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T10:09:36.362 INFO:teuthology.orchestra.run.trial072.stderr:done 2026-02-25T10:09:36.364 DEBUG:teuthology.orchestra.run.trial072:> test -d /sys/firmware/efi 2026-02-25T10:09:36.381 DEBUG:teuthology.orchestra.run.trial072:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T10:09:36.446 INFO:teuthology.orchestra.run.trial072.stdout:/boot/efi/EFI/BOOT 2026-02-25T10:09:36.447 INFO:teuthology.orchestra.run.trial072.stdout:/boot/efi/EFI/centos 2026-02-25T10:09:36.448 DEBUG:teuthology.orchestra.run.trial072:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T10:09:36.513 INFO:teuthology.orchestra.run.trial072.stdout:no 2026-02-25T10:09:36.513 DEBUG:teuthology.orchestra.run.trial072:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T10:09:36.579 INFO:teuthology.orchestra.run.trial072.stdout:yes 2026-02-25T10:09:36.579 DEBUG:teuthology.orchestra.run.trial072:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T10:09:36.643 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial072 2026-02-25T10:09:36.643 DEBUG:teuthology.orchestra.run.trial072:> sudo bash -c ' 2026-02-25T10:09:36.643 DEBUG:teuthology.orchestra.run.trial072:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T10:09:36.644 DEBUG:teuthology.orchestra.run.trial072:> ' 2026-02-25T10:09:51.209 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:51.230 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:51.419 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:51.419 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T10:09:51.419 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T10:09:51.419 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T10:09:51.476 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T10:09:51.476 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:09:51.476 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:09:51.476 INFO:teuthology.orchestra.run.trial068.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T10:09:51.476 INFO:teuthology.orchestra.run.trial068.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T10:09:51.476 INFO:teuthology.orchestra.run.trial068.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T10:09:51.477 INFO:teuthology.orchestra.run.trial068.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T10:09:51.477 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:09:51.477 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:09:51.538 DEBUG:teuthology.orchestra.run.trial068:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T10:09:51.567 INFO:teuthology.orchestra.run.trial068.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T10:09:51.567 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T10:09:51.567 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T10:09:51.567 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T10:09:51.567 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T10:09:51.567 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T10:09:51.567 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial068.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T10:09:51.567 DEBUG:teuthology.orchestra.run.trial068:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T10:09:51.634 INFO:teuthology.orchestra.run.trial068.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T10:09:51.635 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T10:09:51.636 DEBUG:teuthology.orchestra.run.trial068:> sudo rpm -qi grub2-tools 2026-02-25T10:09:51.705 INFO:teuthology.orchestra.run.trial068.stdout:Name : grub2-tools 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Epoch : 1 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Version : 2.06 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Release : 120.el9 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Architecture: x86_64 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Group : Unspecified 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Size : 9108611 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:License : GPLv3+ 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Packager : builder@centos.org 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Vendor : CentOS 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Summary : Support tools for GRUB. 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:Description : 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:hardware devices. 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:09:51.706 INFO:teuthology.orchestra.run.trial068.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T10:09:51.707 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T10:09:51.707 INFO:teuthology.task.kernel:Updating grub on trial068 to boot 5.14.0-686.el9.x86_64 2026-02-25T10:09:51.707 DEBUG:teuthology.orchestra.run.trial068:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T10:09:51.772 INFO:teuthology.orchestra.run.trial068.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T10:09:51.774 DEBUG:teuthology.orchestra.run.trial068:> sudo command -v grubby && echo yes || echo no 2026-02-25T10:09:51.837 INFO:teuthology.orchestra.run.trial068.stdout:/sbin/grubby 2026-02-25T10:09:51.838 INFO:teuthology.orchestra.run.trial068.stdout:yes 2026-02-25T10:09:51.839 DEBUG:teuthology.orchestra.run.trial068:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T10:09:51.902 INFO:teuthology.orchestra.run.trial068.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T10:09:51.903 DEBUG:teuthology.orchestra.run.trial068:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T10:09:51.976 DEBUG:teuthology.orchestra.run.trial068:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T10:09:52.186 INFO:teuthology.orchestra.run.trial068.stderr:Generating grub configuration file ... 2026-02-25T10:09:52.253 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T10:09:52.260 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T10:09:52.510 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T10:09:52.516 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T10:09:52.550 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T10:09:52.556 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T10:09:52.587 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T10:09:52.594 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T10:09:52.617 INFO:teuthology.orchestra.run.trial068.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T10:09:52.623 INFO:teuthology.orchestra.run.trial068.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T10:09:52.709 INFO:teuthology.orchestra.run.trial068.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T10:09:52.717 INFO:teuthology.orchestra.run.trial068.stderr:done 2026-02-25T10:09:52.719 DEBUG:teuthology.orchestra.run.trial068:> test -d /sys/firmware/efi 2026-02-25T10:09:52.736 DEBUG:teuthology.orchestra.run.trial068:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T10:09:52.801 INFO:teuthology.orchestra.run.trial068.stdout:/boot/efi/EFI/BOOT 2026-02-25T10:09:52.801 INFO:teuthology.orchestra.run.trial068.stdout:/boot/efi/EFI/centos 2026-02-25T10:09:52.803 DEBUG:teuthology.orchestra.run.trial068:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T10:09:52.871 INFO:teuthology.orchestra.run.trial068.stdout:no 2026-02-25T10:09:52.871 DEBUG:teuthology.orchestra.run.trial068:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T10:09:52.937 INFO:teuthology.orchestra.run.trial068.stdout:yes 2026-02-25T10:09:52.938 DEBUG:teuthology.orchestra.run.trial068:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T10:09:53.001 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial068 2026-02-25T10:09:53.001 DEBUG:teuthology.orchestra.run.trial068:> sudo bash -c ' 2026-02-25T10:09:53.002 DEBUG:teuthology.orchestra.run.trial068:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T10:09:53.002 DEBUG:teuthology.orchestra.run.trial068:> ' 2026-02-25T10:10:06.687 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2026-02-25T10:10:06.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial072.front.sepia.ceph.com' 2026-02-25T10:10:06.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:10:23.045 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T10:10:23.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-25T10:10:23.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:11:06.733 DEBUG:teuthology.orchestra.remote:timed out 2026-02-25T10:11:15.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial072.front.sepia.ceph.com' 2026-02-25T10:11:15.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:11:15.877 DEBUG:teuthology.orchestra.run.trial072:> true 2026-02-25T10:11:16.027 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial072.front.sepia.ceph.com' 2026-02-25T10:11:16.028 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-686.el9.x86_64"... 2026-02-25T10:11:16.028 DEBUG:teuthology.orchestra.run.trial072:> uname -r 2026-02-25T10:11:16.084 INFO:teuthology.orchestra.run.trial072.stdout:5.14.0-686.el9.x86_64 2026-02-25T10:11:16.084 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T10:11:16.084 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T10:11:16.085 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T10:11:17.086 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2026-02-25T10:11:17.086 DEBUG:teuthology.orchestra.run.trial072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T10:11:17.117 INFO:teuthology.orchestra.run.trial072.stdout:ttyS1 2026-02-25T10:11:17.137 DEBUG:teuthology.parallel:result is None 2026-02-25T10:11:23.048 DEBUG:teuthology.orchestra.remote:timed out 2026-02-25T10:11:32.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-25T10:11:32.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:11:32.187 DEBUG:teuthology.orchestra.run.trial068:> true 2026-02-25T10:11:32.333 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-25T10:11:32.333 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T10:11:32.333 DEBUG:teuthology.orchestra.run.trial068:> uname -r 2026-02-25T10:11:32.390 INFO:teuthology.orchestra.run.trial068.stdout:5.14.0-686.el9.x86_64 2026-02-25T10:11:32.390 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T10:11:32.390 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T10:11:32.390 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T10:11:33.392 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T10:11:33.392 DEBUG:teuthology.orchestra.run.trial068:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T10:11:33.422 INFO:teuthology.orchestra.run.trial068.stdout:ttyS1 2026-02-25T10:11:33.444 DEBUG:teuthology.parallel:result is None 2026-02-25T10:11:33.444 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T10:11:33.457 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T10:11:33.457 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T10:11:33.486 DEBUG:teuthology.orchestra.run.trial072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T10:11:33.505 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T10:11:33.517 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T10:11:33.530 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T10:11:33.531 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T10:11:33.543 DEBUG:teuthology.orchestra.run.trial072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T10:11:33.568 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T10:11:33.580 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T10:11:33.581 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T10:11:33.600 DEBUG:teuthology.orchestra.run.trial072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T10:11:33.622 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T10:11:33.626 INFO:teuthology.orchestra.run.trial072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T10:11:33.630 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T10:11:33.635 INFO:teuthology.orchestra.run.trial072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T10:11:33.637 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T10:11:33.648 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T10:11:33.649 DEBUG:teuthology.orchestra.run.trial068:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T10:11:33.674 DEBUG:teuthology.orchestra.run.trial072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T10:11:33.701 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T10:11:33.718 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T10:11:33.719 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T10:11:33.739 DEBUG:teuthology.orchestra.run.trial072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T10:11:33.758 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T10:11:33.884 DEBUG:teuthology.orchestra.run.trial068:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T10:11:33.909 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T10:11:34.046 DEBUG:teuthology.orchestra.run.trial068:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T10:11:34.071 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:11:34.071 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T10:11:34.133 DEBUG:teuthology.orchestra.run.trial072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T10:11:34.223 DEBUG:teuthology.orchestra.run.trial072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T10:11:34.249 DEBUG:teuthology.orchestra.run.trial072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T10:11:34.375 DEBUG:teuthology.orchestra.run.trial072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T10:11:34.398 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:11:34.398 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T10:11:34.462 DEBUG:teuthology.orchestra.run.trial068:> sudo service rsyslog restart 2026-02-25T10:11:34.464 DEBUG:teuthology.orchestra.run.trial072:> sudo service rsyslog restart 2026-02-25T10:11:34.491 INFO:teuthology.orchestra.run.trial068.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T10:11:34.528 INFO:teuthology.orchestra.run.trial072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T10:11:35.016 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T10:11:35.032 INFO:teuthology.task.internal:Starting timer... 2026-02-25T10:11:35.032 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T10:11:35.054 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T10:11:35.072 DEBUG:teuthology.orchestra.run.trial068:> sudo service auditd rotate 2026-02-25T10:11:35.100 INFO:teuthology.orchestra.run.trial068.stdout:Rotating logs: 2026-02-25T10:11:35.102 DEBUG:teuthology.orchestra.run.trial072:> sudo service auditd rotate 2026-02-25T10:11:35.132 INFO:teuthology.orchestra.run.trial072.stdout:Rotating logs: 2026-02-25T10:11:35.134 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T10:11:35.134 DEBUG:teuthology.orchestra.run.trial068:> /usr/sbin/getenforce 2026-02-25T10:11:35.158 INFO:teuthology.orchestra.run.trial068.stdout:Permissive 2026-02-25T10:11:35.158 DEBUG:teuthology.orchestra.run.trial072:> /usr/sbin/getenforce 2026-02-25T10:11:35.190 INFO:teuthology.orchestra.run.trial072.stdout:Permissive 2026-02-25T10:11:35.190 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial068.front.sepia.ceph.com': 'permissive', 'ubuntu@trial072.front.sepia.ceph.com': 'permissive'} 2026-02-25T10:11:35.191 DEBUG:teuthology.orchestra.run.trial068:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T10:11:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:11:35.224 DEBUG:teuthology.orchestra.run.trial072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T10:11:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:11:35.256 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T10:11:35.256 DEBUG:teuthology.orchestra.run.trial068:> sudo /usr/sbin/setenforce permissive 2026-02-25T10:11:35.287 DEBUG:teuthology.orchestra.run.trial072:> sudo /usr/sbin/setenforce permissive 2026-02-25T10:11:35.319 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T10:11:35.335 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T10:11:35.348 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T10:11:35.349 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial068.front.sepia.ceph.com,trial072.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T10:14:26.118 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial068.front.sepia.ceph.com'), Remote(name='ubuntu@trial072.front.sepia.ceph.com')] 2026-02-25T10:14:26.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-25T10:14:26.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:14:26.226 DEBUG:teuthology.orchestra.run.trial068:> true 2026-02-25T10:14:26.302 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-25T10:14:26.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial072.front.sepia.ceph.com' 2026-02-25T10:14:26.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T10:14:26.398 DEBUG:teuthology.orchestra.run.trial072:> true 2026-02-25T10:14:26.475 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial072.front.sepia.ceph.com' 2026-02-25T10:14:26.475 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T10:14:26.500 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T10:14:26.500 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T10:14:26.501 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T10:14:26.508 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T10:14:26.508 DEBUG:teuthology.orchestra.run.trial072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T10:14:26.534 INFO:teuthology.orchestra.run.trial068.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T10:14:26.542 INFO:teuthology.orchestra.run.trial072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T10:14:26.546 INFO:teuthology.orchestra.run.trial068.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T10:14:26.553 INFO:teuthology.orchestra.run.trial072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T10:14:26.567 INFO:teuthology.orchestra.run.trial068.stderr:sudo: ntpd: command not found 2026-02-25T10:14:26.573 INFO:teuthology.orchestra.run.trial072.stderr:sudo: ntpd: command not found 2026-02-25T10:14:26.577 INFO:teuthology.orchestra.run.trial068.stdout:506 Cannot talk to daemon 2026-02-25T10:14:26.583 INFO:teuthology.orchestra.run.trial072.stdout:506 Cannot talk to daemon 2026-02-25T10:14:26.589 INFO:teuthology.orchestra.run.trial068.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T10:14:26.594 INFO:teuthology.orchestra.run.trial072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T10:14:26.600 INFO:teuthology.orchestra.run.trial068.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T10:14:26.606 INFO:teuthology.orchestra.run.trial072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T10:14:26.646 INFO:teuthology.orchestra.run.trial068.stderr:bash: line 1: ntpq: command not found 2026-02-25T10:14:26.648 INFO:teuthology.orchestra.run.trial068.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T10:14:26.648 INFO:teuthology.orchestra.run.trial068.stdout:=============================================================================== 2026-02-25T10:14:26.651 INFO:teuthology.orchestra.run.trial072.stderr:bash: line 1: ntpq: command not found 2026-02-25T10:14:26.652 INFO:teuthology.orchestra.run.trial072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T10:14:26.653 INFO:teuthology.orchestra.run.trial072.stdout:=============================================================================== 2026-02-25T10:14:26.654 INFO:teuthology.run_tasks:Running task install... 2026-02-25T10:14:26.668 DEBUG:teuthology.task.install:project ceph 2026-02-25T10:14:26.669 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T10:14:26.669 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T10:14:26.669 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T10:14:26.671 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T10:14:26.672 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-25T10:14:26.672 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T10:14:26.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:14:26.674 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T10:14:26.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:14:26.844 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T10:14:26.844 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T10:14:26.856 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T10:14:26.856 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T10:14:27.069 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T10:14:27.070 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:14:27.070 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T10:14:27.095 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T10:14:27.095 DEBUG:teuthology.orchestra.run.trial068:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T10:14:27.106 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T10:14:27.106 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:14:27.106 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T10:14:27.132 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T10:14:27.132 DEBUG:teuthology.orchestra.run.trial072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T10:14:27.159 DEBUG:teuthology.orchestra.run.trial068:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T10:14:27.197 DEBUG:teuthology.orchestra.run.trial072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T10:14:27.232 DEBUG:teuthology.orchestra.run.trial068:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T10:14:27.272 DEBUG:teuthology.orchestra.run.trial072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T10:14:27.296 INFO:teuthology.orchestra.run.trial068.stdout:check_obsoletes = 1 2026-02-25T10:14:27.298 DEBUG:teuthology.orchestra.run.trial068:> sudo yum clean all 2026-02-25T10:14:27.336 INFO:teuthology.orchestra.run.trial072.stdout:check_obsoletes = 1 2026-02-25T10:14:27.338 DEBUG:teuthology.orchestra.run.trial072:> sudo yum clean all 2026-02-25T10:14:27.447 INFO:teuthology.orchestra.run.trial068.stdout:41 files removed 2026-02-25T10:14:27.461 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-radosgw 2026-02-25T10:14:27.485 INFO:teuthology.orchestra.run.trial072.stdout:41 files removed 2026-02-25T10:14:27.499 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-radosgw 2026-02-25T10:14:28.030 INFO:teuthology.orchestra.run.trial072.stdout:ceph packages for x86_64 244 kB/s | 94 kB 00:00 2026-02-25T10:14:28.403 INFO:teuthology.orchestra.run.trial072.stdout:ceph noarch packages 71 kB/s | 24 kB 00:00 2026-02-25T10:14:28.737 INFO:teuthology.orchestra.run.trial072.stdout:ceph source packages 6.7 kB/s | 2.1 kB 00:00 2026-02-25T10:14:29.539 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-25T10:14:30.040 INFO:teuthology.orchestra.run.trial068.stdout:ceph packages for x86_64 39 kB/s | 94 kB 00:02 2026-02-25T10:14:30.487 INFO:teuthology.orchestra.run.trial068.stdout:ceph noarch packages 58 kB/s | 24 kB 00:00 2026-02-25T10:14:30.854 INFO:teuthology.orchestra.run.trial068.stdout:ceph source packages 6.0 kB/s | 2.1 kB 00:00 2026-02-25T10:14:31.644 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-25T10:14:32.212 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-25T10:14:34.825 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-25T10:14:36.346 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 8.0 MB 00:01 2026-02-25T10:14:37.875 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 20 kB 00:00 2026-02-25T10:14:38.244 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 8.0 MB 00:00 2026-02-25T10:14:38.621 INFO:teuthology.orchestra.run.trial072.stdout:Extra Packages for Enterprise Linux 30 MB/s | 20 MB 00:00 2026-02-25T10:14:39.147 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - Extras packages 127 kB/s | 20 kB 00:00 2026-02-25T10:14:40.105 INFO:teuthology.orchestra.run.trial068.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-25T10:14:41.773 INFO:teuthology.orchestra.run.trial072.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T10:14:42.986 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repository Size 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout:Upgrading: 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T10:14:42.988 INFO:teuthology.orchestra.run.trial072.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout:Installing dependencies: 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T10:14:42.989 INFO:teuthology.orchestra.run.trial072.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T10:14:42.990 INFO:teuthology.orchestra.run.trial072.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout:Installing weak dependencies: 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T10:14:42.991 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout:Install 35 Packages 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout:Upgrade 2 Packages 2026-02-25T10:14:42.992 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:42.993 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 61 M 2026-02-25T10:14:42.993 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:14:43.369 INFO:teuthology.orchestra.run.trial068.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T10:14:43.981 INFO:teuthology.orchestra.run.trial072.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 6.7 MB/s | 5.1 MB 00:00 2026-02-25T10:14:44.026 INFO:teuthology.orchestra.run.trial072.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 566 kB/s | 25 kB 00:00 2026-02-25T10:14:44.050 INFO:teuthology.orchestra.run.trial072.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 1.5 MB/s | 37 kB 00:00 2026-02-25T10:14:44.079 INFO:teuthology.orchestra.run.trial072.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 909 kB/s | 26 kB 00:00 2026-02-25T10:14:44.190 INFO:teuthology.orchestra.run.trial072.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 7.6 MB/s | 860 kB 00:00 2026-02-25T10:14:44.239 INFO:teuthology.orchestra.run.trial072.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 5.2 MB/s | 263 kB 00:00 2026-02-25T10:14:44.584 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:14:44.586 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout:Upgrading: 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T10:14:44.587 INFO:teuthology.orchestra.run.trial068.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout:Installing weak dependencies: 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout:Install 35 Packages 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout:Upgrade 2 Packages 2026-02-25T10:14:44.588 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:44.589 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 61 M 2026-02-25T10:14:44.589 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:14:44.687 INFO:teuthology.orchestra.run.trial072.stdout:(7/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_6 13 MB/s | 5.6 MB 00:00 2026-02-25T10:14:44.956 INFO:teuthology.orchestra.run.trial072.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 168 kB/s | 45 kB 00:00 2026-02-25T10:14:45.114 INFO:teuthology.orchestra.run.trial072.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 1.1 MB/s | 185 kB 00:00 2026-02-25T10:14:45.164 INFO:teuthology.orchestra.run.trial072.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 4.0 MB/s | 204 kB 00:00 2026-02-25T10:14:45.217 INFO:teuthology.orchestra.run.trial072.stdout:(11/37): python3-rados-20.3.0-5338.g6aeb6975.el 6.0 MB/s | 324 kB 00:00 2026-02-25T10:14:45.291 INFO:teuthology.orchestra.run.trial072.stdout:(12/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 4.1 MB/s | 306 kB 00:00 2026-02-25T10:14:45.342 INFO:teuthology.orchestra.run.trial072.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 1.9 MB/s | 100 kB 00:00 2026-02-25T10:14:45.372 INFO:teuthology.orchestra.run.trial072.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 1.0 MB/s | 33 kB 00:00 2026-02-25T10:14:45.417 INFO:teuthology.orchestra.run.trial072.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 26 MB/s | 1.2 MB 00:00 2026-02-25T10:14:45.427 INFO:teuthology.orchestra.run.trial072.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 8.9 MB/s | 93 kB 00:00 2026-02-25T10:14:45.992 INFO:teuthology.orchestra.run.trial072.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 184 kB/s | 104 kB 00:00 2026-02-25T10:14:46.153 INFO:teuthology.orchestra.run.trial072.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 1.0 MB/s | 164 kB 00:00 2026-02-25T10:14:46.236 INFO:teuthology.orchestra.run.trial072.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2026-02-25T10:14:46.320 INFO:teuthology.orchestra.run.trial072.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 546 kB/s | 45 kB 00:00 2026-02-25T10:14:46.401 INFO:teuthology.orchestra.run.trial072.stdout:(21/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9 3.5 MB/s | 11 MB 00:03 2026-02-25T10:14:46.506 INFO:teuthology.orchestra.run.trial072.stdout:(22/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 6.3 MB/s | 21 MB 00:03 2026-02-25T10:14:46.511 INFO:teuthology.orchestra.run.trial072.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2026-02-25T10:14:46.594 INFO:teuthology.orchestra.run.trial072.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 2.8 MB/s | 240 kB 00:00 2026-02-25T10:14:46.661 INFO:teuthology.orchestra.run.trial068.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 2.9 MB/s | 5.1 MB 00:01 2026-02-25T10:14:46.680 INFO:teuthology.orchestra.run.trial072.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 769 kB/s | 66 kB 00:00 2026-02-25T10:14:46.705 INFO:teuthology.orchestra.run.trial068.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 566 kB/s | 25 kB 00:00 2026-02-25T10:14:46.752 INFO:teuthology.orchestra.run.trial068.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 806 kB/s | 37 kB 00:00 2026-02-25T10:14:46.796 INFO:teuthology.orchestra.run.trial068.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 583 kB/s | 26 kB 00:00 2026-02-25T10:14:46.869 INFO:teuthology.orchestra.run.trial072.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.6 MB/s | 308 kB 00:00 2026-02-25T10:14:46.919 INFO:teuthology.orchestra.run.trial072.stdout:(27/37): qatlib-service-25.08.0-2.el9.x86_64.rp 91 kB/s | 37 kB 00:00 2026-02-25T10:14:46.949 INFO:teuthology.orchestra.run.trial068.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 5.5 MB/s | 860 kB 00:00 2026-02-25T10:14:46.994 INFO:teuthology.orchestra.run.trial072.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 329 kB/s | 25 kB 00:00 2026-02-25T10:14:46.999 INFO:teuthology.orchestra.run.trial072.stdout:(29/37): libarrow-9.0.0-15.el9.x86_64.rpm 34 MB/s | 4.4 MB 00:00 2026-02-25T10:14:47.001 INFO:teuthology.orchestra.run.trial068.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 4.9 MB/s | 263 kB 00:00 2026-02-25T10:14:47.019 INFO:teuthology.orchestra.run.trial072.stdout:(30/37): liboath-2.6.12-1.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2026-02-25T10:14:47.024 INFO:teuthology.orchestra.run.trial072.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2026-02-25T10:14:47.047 INFO:teuthology.orchestra.run.trial072.stdout:(32/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 29 MB/s | 838 kB 00:00 2026-02-25T10:14:47.049 INFO:teuthology.orchestra.run.trial072.stdout:(33/37): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2026-02-25T10:14:47.058 INFO:teuthology.orchestra.run.trial072.stdout:(34/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 445 kB/s | 292 kB 00:00 2026-02-25T10:14:47.078 INFO:teuthology.orchestra.run.trial072.stdout:(35/37): thrift-0.15.0-4.el9.x86_64.rpm 50 MB/s | 1.6 MB 00:00 2026-02-25T10:14:47.454 INFO:teuthology.orchestra.run.trial072.stdout:(36/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 6.8 MB/s | 2.7 MB 00:00 2026-02-25T10:14:47.684 INFO:teuthology.orchestra.run.trial072.stdout:(37/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 5.2 MB/s | 3.3 MB 00:00 2026-02-25T10:14:47.688 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:14:47.688 INFO:teuthology.orchestra.run.trial072.stdout:Total 13 MB/s | 61 MB 00:04 2026-02-25T10:14:47.727 INFO:teuthology.orchestra.run.trial068.stdout:(7/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_6 7.8 MB/s | 5.6 MB 00:00 2026-02-25T10:14:47.769 INFO:teuthology.orchestra.run.trial068.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 1.1 MB/s | 45 kB 00:00 2026-02-25T10:14:47.784 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:14:47.803 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:14:47.803 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:14:47.813 INFO:teuthology.orchestra.run.trial068.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 4.1 MB/s | 185 kB 00:00 2026-02-25T10:14:47.862 INFO:teuthology.orchestra.run.trial068.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 4.1 MB/s | 204 kB 00:00 2026-02-25T10:14:47.912 INFO:teuthology.orchestra.run.trial068.stdout:(11/37): python3-rados-20.3.0-5338.g6aeb6975.el 6.3 MB/s | 324 kB 00:00 2026-02-25T10:14:47.963 INFO:teuthology.orchestra.run.trial068.stdout:(12/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 5.9 MB/s | 306 kB 00:00 2026-02-25T10:14:48.010 INFO:teuthology.orchestra.run.trial068.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 2.1 MB/s | 100 kB 00:00 2026-02-25T10:14:48.052 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:14:48.053 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:14:48.099 INFO:teuthology.orchestra.run.trial068.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 376 kB/s | 33 kB 00:00 2026-02-25T10:14:48.207 INFO:teuthology.orchestra.run.trial068.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2026-02-25T10:14:48.229 INFO:teuthology.orchestra.run.trial068.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 4.1 MB/s | 93 kB 00:00 2026-02-25T10:14:48.248 INFO:teuthology.orchestra.run.trial068.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 5.5 MB/s | 104 kB 00:00 2026-02-25T10:14:48.257 INFO:teuthology.orchestra.run.trial068.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 18 MB/s | 164 kB 00:00 2026-02-25T10:14:48.264 INFO:teuthology.orchestra.run.trial068.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 22 MB/s | 160 kB 00:00 2026-02-25T10:14:48.269 INFO:teuthology.orchestra.run.trial068.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 9.7 MB/s | 45 kB 00:00 2026-02-25T10:14:48.280 INFO:teuthology.orchestra.run.trial068.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 58 MB/s | 662 kB 00:00 2026-02-25T10:14:48.287 INFO:teuthology.orchestra.run.trial068.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 42 MB/s | 292 kB 00:00 2026-02-25T10:14:48.295 INFO:teuthology.orchestra.run.trial068.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 30 MB/s | 240 kB 00:00 2026-02-25T10:14:48.300 INFO:teuthology.orchestra.run.trial068.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 7.8 MB/s | 37 kB 00:00 2026-02-25T10:14:48.305 INFO:teuthology.orchestra.run.trial068.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 14 MB/s | 66 kB 00:00 2026-02-25T10:14:48.346 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:14:48.471 INFO:teuthology.orchestra.run.trial068.stdout:(26/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 5.8 MB/s | 21 MB 00:03 2026-02-25T10:14:48.488 INFO:teuthology.orchestra.run.trial072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T10:14:48.494 INFO:teuthology.orchestra.run.trial072.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T10:14:48.543 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T10:14:48.544 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T10:14:48.557 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T10:14:48.559 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T10:14:48.560 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T10:14:48.587 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T10:14:48.591 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T10:14:48.602 INFO:teuthology.orchestra.run.trial072.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T10:14:48.604 INFO:teuthology.orchestra.run.trial072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T10:14:48.607 INFO:teuthology.orchestra.run.trial072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T10:14:48.611 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T10:14:48.623 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T10:14:48.629 INFO:teuthology.orchestra.run.trial068.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 949 kB/s | 308 kB 00:00 2026-02-25T10:14:48.635 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T10:14:48.636 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T10:14:48.658 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T10:14:48.661 INFO:teuthology.orchestra.run.trial072.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T10:14:48.672 INFO:teuthology.orchestra.run.trial068.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 579 kB/s | 25 kB 00:00 2026-02-25T10:14:48.693 INFO:teuthology.orchestra.run.trial072.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T10:14:48.697 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T10:14:48.703 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T10:14:48.705 INFO:teuthology.orchestra.run.trial072.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T10:14:48.710 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T10:14:48.717 INFO:teuthology.orchestra.run.trial068.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2026-02-25T10:14:48.771 INFO:teuthology.orchestra.run.trial068.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2026-02-25T10:14:48.826 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T10:14:48.827 INFO:teuthology.orchestra.run.trial072.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T10:14:48.853 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T10:14:48.855 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T10:14:48.862 INFO:teuthology.orchestra.run.trial072.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T10:14:48.889 INFO:teuthology.orchestra.run.trial068.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 6.9 MB/s | 838 kB 00:00 2026-02-25T10:14:48.940 INFO:teuthology.orchestra.run.trial068.stdout:(32/37): libarrow-9.0.0-15.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2026-02-25T10:14:48.953 INFO:teuthology.orchestra.run.trial068.stdout:(33/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9 2.8 MB/s | 11 MB 00:04 2026-02-25T10:14:48.954 INFO:teuthology.orchestra.run.trial068.stdout:(34/37): re2-20211101-20.el9.x86_64.rpm 2.9 MB/s | 191 kB 00:00 2026-02-25T10:14:48.992 INFO:teuthology.orchestra.run.trial068.stdout:(35/37): thrift-0.15.0-4.el9.x86_64.rpm 30 MB/s | 1.6 MB 00:00 2026-02-25T10:14:49.110 INFO:teuthology.orchestra.run.trial072.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T10:14:49.111 INFO:teuthology.orchestra.run.trial072.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T10:14:49.125 INFO:teuthology.orchestra.run.trial068.stdout:(36/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 16 MB/s | 2.7 MB 00:00 2026-02-25T10:14:49.127 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T10:14:49.131 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T10:14:49.138 INFO:teuthology.orchestra.run.trial072.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T10:14:49.140 INFO:teuthology.orchestra.run.trial072.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T10:14:49.141 INFO:teuthology.orchestra.run.trial072.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T10:14:49.160 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T10:14:49.211 INFO:teuthology.orchestra.run.trial068.stdout:(37/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 13 MB/s | 3.3 MB 00:00 2026-02-25T10:14:49.214 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:14:49.215 INFO:teuthology.orchestra.run.trial068.stdout:Total 13 MB/s | 61 MB 00:04 2026-02-25T10:14:49.283 INFO:teuthology.orchestra.run.trial072.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T10:14:49.286 INFO:teuthology.orchestra.run.trial072.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T10:14:49.316 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:14:49.334 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:14:49.334 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:14:49.566 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T10:14:49.567 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T10:14:49.567 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:49.581 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:14:49.581 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:14:49.611 INFO:teuthology.orchestra.run.trial072.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T10:14:49.614 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T10:14:49.872 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:14:50.018 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T10:14:50.023 INFO:teuthology.orchestra.run.trial068.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T10:14:50.072 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T10:14:50.073 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T10:14:50.086 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T10:14:50.090 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T10:14:50.090 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T10:14:50.119 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T10:14:50.121 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T10:14:50.130 INFO:teuthology.orchestra.run.trial068.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T10:14:50.133 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T10:14:50.136 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T10:14:50.140 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T10:14:50.151 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T10:14:50.164 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T10:14:50.165 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T10:14:50.187 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T10:14:50.191 INFO:teuthology.orchestra.run.trial068.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T10:14:50.222 INFO:teuthology.orchestra.run.trial068.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T10:14:50.226 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T10:14:50.233 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T10:14:50.234 INFO:teuthology.orchestra.run.trial068.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T10:14:50.239 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T10:14:50.357 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T10:14:50.358 INFO:teuthology.orchestra.run.trial068.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T10:14:50.386 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T10:14:50.388 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T10:14:50.394 INFO:teuthology.orchestra.run.trial068.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T10:14:50.555 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T10:14:50.557 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T10:14:50.649 INFO:teuthology.orchestra.run.trial068.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T10:14:50.651 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T10:14:50.667 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T10:14:50.671 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T10:14:50.676 INFO:teuthology.orchestra.run.trial068.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T10:14:50.678 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T10:14:50.679 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T10:14:50.698 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T10:14:50.813 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T10:14:50.815 INFO:teuthology.orchestra.run.trial068.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T10:14:50.818 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T10:14:50.822 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T10:14:50.861 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T10:14:50.861 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T10:14:50.862 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T10:14:50.862 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:50.865 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T10:14:51.089 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T10:14:51.089 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T10:14:51.090 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:51.134 INFO:teuthology.orchestra.run.trial068.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T10:14:51.136 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T10:14:52.108 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T10:14:52.111 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T10:14:52.384 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T10:14:52.388 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T10:14:52.430 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T10:14:52.430 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T10:14:52.430 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T10:14:52.430 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:52.434 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T10:14:55.429 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T10:14:55.429 INFO:teuthology.orchestra.run.trial072.stdout:skipping the directory /sys 2026-02-25T10:14:55.429 INFO:teuthology.orchestra.run.trial072.stdout:skipping the directory /proc 2026-02-25T10:14:55.429 INFO:teuthology.orchestra.run.trial072.stdout:skipping the directory /mnt 2026-02-25T10:14:55.429 INFO:teuthology.orchestra.run.trial072.stdout:skipping the directory /var/tmp 2026-02-25T10:14:55.430 INFO:teuthology.orchestra.run.trial072.stdout:skipping the directory /home 2026-02-25T10:14:55.430 INFO:teuthology.orchestra.run.trial072.stdout:skipping the directory /root 2026-02-25T10:14:55.430 INFO:teuthology.orchestra.run.trial072.stdout:skipping the directory /tmp 2026-02-25T10:14:55.430 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:55.920 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T10:14:55.943 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T10:14:55.943 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:14:55.943 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T10:14:55.943 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T10:14:55.944 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T10:14:55.944 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:55.945 INFO:teuthology.orchestra.run.trial072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T10:14:55.957 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T10:14:55.957 INFO:teuthology.orchestra.run.trial072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T10:14:56.013 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T10:14:56.013 INFO:teuthology.orchestra.run.trial072.stdout:Creating group 'qat' with GID 992. 2026-02-25T10:14:56.013 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T10:14:56.294 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T10:14:56.295 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T10:14:56.297 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T10:14:56.298 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T10:14:56.362 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout:Upgraded: 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.363 INFO:teuthology.orchestra.run.trial072.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T10:14:56.364 INFO:teuthology.orchestra.run.trial072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.365 INFO:teuthology.orchestra.run.trial072.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:56.366 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:14:56.402 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-test 2026-02-25T10:14:56.718 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:15 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:14:56.802 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:14:56.803 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:14:56.803 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repository Size 2026-02-25T10:14:56.803 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:14:56.803 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:14:56.804 INFO:teuthology.orchestra.run.trial072.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T10:14:56.804 INFO:teuthology.orchestra.run.trial072.stdout:Installing dependencies: 2026-02-25T10:14:56.804 INFO:teuthology.orchestra.run.trial072.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T10:14:56.804 INFO:teuthology.orchestra.run.trial072.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T10:14:56.804 INFO:teuthology.orchestra.run.trial072.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T10:14:56.804 INFO:teuthology.orchestra.run.trial072.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T10:14:56.804 INFO:teuthology.orchestra.run.trial072.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout:Install 12 Packages 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:14:56.805 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 132 M 2026-02-25T10:14:56.806 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 988 M 2026-02-25T10:14:56.806 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /sys 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /proc 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /mnt 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /var/tmp 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /home 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /root 2026-02-25T10:14:56.924 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /tmp 2026-02-25T10:14:56.925 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:57.311 INFO:teuthology.orchestra.run.trial072.stdout:(1/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 503 kB/s | 153 kB 00:00 2026-02-25T10:14:57.436 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T10:14:57.460 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T10:14:57.460 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:14:57.460 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T10:14:57.460 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T10:14:57.460 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T10:14:57.460 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:57.462 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T10:14:57.475 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T10:14:57.475 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T10:14:57.530 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T10:14:57.530 INFO:teuthology.orchestra.run.trial068.stdout:Creating group 'qat' with GID 992. 2026-02-25T10:14:57.530 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:57.788 INFO:teuthology.orchestra.run.trial072.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 48 kB/s | 23 kB 00:00 2026-02-25T10:14:57.819 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T10:14:57.820 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T10:14:57.821 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T10:14:57.822 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T10:14:57.887 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T10:14:57.887 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout:Upgraded: 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.888 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T10:14:57.889 INFO:teuthology.orchestra.run.trial068.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T10:14:57.890 INFO:teuthology.orchestra.run.trial068.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T10:14:57.891 INFO:teuthology.orchestra.run.trial068.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T10:14:57.892 INFO:teuthology.orchestra.run.trial068.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T10:14:57.892 INFO:teuthology.orchestra.run.trial068.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T10:14:57.892 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:57.892 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:14:57.928 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-test 2026-02-25T10:14:58.025 INFO:teuthology.orchestra.run.trial072.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 397 kB/s | 94 kB 00:00 2026-02-25T10:14:58.028 INFO:teuthology.orchestra.run.trial072.stdout:(4/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.1 MB/s | 2.1 MB 00:01 2026-02-25T10:14:58.248 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:15 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:14:58.333 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:14:58.334 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:14:58.334 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-25T10:14:58.334 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T10:14:58.335 INFO:teuthology.orchestra.run.trial068.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:14:58.336 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:14:58.337 INFO:teuthology.orchestra.run.trial068.stdout:Install 12 Packages 2026-02-25T10:14:58.337 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:14:58.337 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 132 M 2026-02-25T10:14:58.337 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 988 M 2026-02-25T10:14:58.337 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:14:58.487 INFO:teuthology.orchestra.run.trial072.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 385 kB/s | 176 kB 00:00 2026-02-25T10:14:58.608 INFO:teuthology.orchestra.run.trial072.stdout:(6/12): libasan8-15.2.1-7.el9.x86_64.rpm 901 kB/s | 525 kB 00:00 2026-02-25T10:14:58.650 INFO:teuthology.orchestra.run.trial068.stdout:(1/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 1.8 MB/s | 153 kB 00:00 2026-02-25T10:14:58.653 INFO:teuthology.orchestra.run.trial072.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-25T10:14:58.727 INFO:teuthology.orchestra.run.trial068.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 297 kB/s | 23 kB 00:00 2026-02-25T10:14:58.773 INFO:teuthology.orchestra.run.trial072.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 2.5 MB/s | 303 kB 00:00 2026-02-25T10:14:58.798 INFO:teuthology.orchestra.run.trial068.stdout:(3/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 9.0 MB/s | 2.1 MB 00:00 2026-02-25T10:14:58.800 INFO:teuthology.orchestra.run.trial068.stdout:(4/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.3 MB/s | 94 kB 00:00 2026-02-25T10:14:58.813 INFO:teuthology.orchestra.run.trial072.stdout:(9/12): protobuf-3.14.0-17.el9.x86_64.rpm 4.9 MB/s | 1.0 MB 00:00 2026-02-25T10:14:58.859 INFO:teuthology.orchestra.run.trial072.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 744 kB/s | 64 kB 00:00 2026-02-25T10:14:59.092 INFO:teuthology.orchestra.run.trial072.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 436 kB/s | 122 kB 00:00 2026-02-25T10:14:59.515 INFO:teuthology.orchestra.run.trial068.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 247 kB/s | 176 kB 00:00 2026-02-25T10:14:59.681 INFO:teuthology.orchestra.run.trial068.stdout:(6/12): libasan8-15.2.1-7.el9.x86_64.rpm 594 kB/s | 525 kB 00:00 2026-02-25T10:14:59.752 INFO:teuthology.orchestra.run.trial068.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 980 kB/s | 233 kB 00:00 2026-02-25T10:14:59.913 INFO:teuthology.orchestra.run.trial068.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2026-02-25T10:14:59.920 INFO:teuthology.orchestra.run.trial068.stdout:(9/12): protobuf-3.14.0-17.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2026-02-25T10:15:00.056 INFO:teuthology.orchestra.run.trial068.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 898 kB/s | 122 kB 00:00 2026-02-25T10:15:00.073 INFO:teuthology.orchestra.run.trial068.stdout:(11/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 399 kB/s | 64 kB 00:00 2026-02-25T10:15:04.026 INFO:teuthology.orchestra.run.trial072.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 18 MB/s | 127 MB 00:07 2026-02-25T10:15:04.029 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:04.029 INFO:teuthology.orchestra.run.trial072.stdout:Total 18 MB/s | 132 MB 00:07 2026-02-25T10:15:04.109 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:04.120 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:04.120 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:04.265 INFO:teuthology.orchestra.run.trial068.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 22 MB/s | 127 MB 00:05 2026-02-25T10:15:04.266 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:04.267 INFO:teuthology.orchestra.run.trial068.stdout:Total 22 MB/s | 132 MB 00:05 2026-02-25T10:15:04.351 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:04.363 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:04.363 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:04.575 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:04.575 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:04.790 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:04.791 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:05.032 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:05.041 INFO:teuthology.orchestra.run.trial072.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T10:15:05.049 INFO:teuthology.orchestra.run.trial072.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T10:15:05.055 INFO:teuthology.orchestra.run.trial072.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T10:15:05.059 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T10:15:05.061 INFO:teuthology.orchestra.run.trial072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T10:15:05.066 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T10:15:05.067 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T10:15:05.085 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T10:15:05.088 INFO:teuthology.orchestra.run.trial072.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T10:15:05.099 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T10:15:05.108 INFO:teuthology.orchestra.run.trial072.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T10:15:05.109 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T10:15:05.247 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:05.256 INFO:teuthology.orchestra.run.trial068.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T10:15:05.264 INFO:teuthology.orchestra.run.trial068.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T10:15:05.270 INFO:teuthology.orchestra.run.trial068.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T10:15:05.275 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T10:15:05.278 INFO:teuthology.orchestra.run.trial068.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T10:15:05.282 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T10:15:05.283 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T10:15:05.299 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T10:15:05.301 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T10:15:05.311 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T10:15:05.319 INFO:teuthology.orchestra.run.trial068.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T10:15:05.320 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T10:15:12.068 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T10:15:12.088 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T10:15:12.264 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T10:15:12.283 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T10:15:12.938 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T10:15:12.938 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T10:15:12.939 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T10:15:12.996 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T10:15:12.996 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:12.996 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:12.996 INFO:teuthology.orchestra.run.trial072.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:12.996 INFO:teuthology.orchestra.run.trial072.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T10:15:12.996 INFO:teuthology.orchestra.run.trial072.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T10:15:12.997 INFO:teuthology.orchestra.run.trial072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T10:15:12.998 INFO:teuthology.orchestra.run.trial072.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T10:15:12.998 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:12.998 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:13.037 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph 2026-02-25T10:15:13.123 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T10:15:13.124 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T10:15:13.183 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T10:15:13.184 INFO:teuthology.orchestra.run.trial068.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T10:15:13.185 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T10:15:13.185 INFO:teuthology.orchestra.run.trial068.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T10:15:13.185 INFO:teuthology.orchestra.run.trial068.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T10:15:13.185 INFO:teuthology.orchestra.run.trial068.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T10:15:13.185 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:13.185 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:13.227 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph 2026-02-25T10:15:13.345 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:32 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:13.433 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout:======================================================================================= 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repository Size 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout:======================================================================================= 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout:Installing dependencies: 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T10:15:13.436 INFO:teuthology.orchestra.run.trial072.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T10:15:13.437 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T10:15:13.438 INFO:teuthology.orchestra.run.trial072.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout:Installing weak dependencies: 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout:======================================================================================= 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout:Install 96 Packages 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 226 M 2026-02-25T10:15:13.439 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 1.5 G 2026-02-25T10:15:13.440 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:13.548 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:30 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:13.635 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T10:15:13.638 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T10:15:13.639 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T10:15:13.640 INFO:teuthology.orchestra.run.trial068.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T10:15:13.641 INFO:teuthology.orchestra.run.trial068.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T10:15:13.642 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T10:15:13.643 INFO:teuthology.orchestra.run.trial068.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T10:15:13.644 INFO:teuthology.orchestra.run.trial068.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T10:15:13.645 INFO:teuthology.orchestra.run.trial068.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout:Installing weak dependencies: 2026-02-25T10:15:13.646 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:13.647 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-25T10:15:13.648 INFO:teuthology.orchestra.run.trial068.stdout:Install 96 Packages 2026-02-25T10:15:13.648 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:13.649 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 226 M 2026-02-25T10:15:13.649 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 1.5 G 2026-02-25T10:15:13.649 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:13.881 INFO:teuthology.orchestra.run.trial072.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 60 kB/s | 6.5 kB 00:00 2026-02-25T10:15:14.040 INFO:teuthology.orchestra.run.trial072.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 3.4 MB/s | 937 kB 00:00 2026-02-25T10:15:14.068 INFO:teuthology.orchestra.run.trial072.stdout:(3/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 6.8 MB/s | 2.0 MB 00:00 2026-02-25T10:15:14.069 INFO:teuthology.orchestra.run.trial072.stdout:(4/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 5.7 MB/s | 163 kB 00:00 2026-02-25T10:15:14.495 INFO:teuthology.orchestra.run.trial072.stdout:(5/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 35 MB/s | 15 MB 00:00 2026-02-25T10:15:14.518 INFO:teuthology.orchestra.run.trial072.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 2.5 MB/s | 57 kB 00:00 2026-02-25T10:15:14.545 INFO:teuthology.orchestra.run.trial072.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 6.6 MB/s | 184 kB 00:00 2026-02-25T10:15:14.649 INFO:teuthology.orchestra.run.trial072.stdout:(8/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 6.3 MB/s | 4.9 MB 00:00 2026-02-25T10:15:15.036 INFO:teuthology.orchestra.run.trial072.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 19 MB/s | 7.4 MB 00:00 2026-02-25T10:15:15.077 INFO:teuthology.orchestra.run.trial072.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 547 kB/s | 22 kB 00:00 2026-02-25T10:15:15.123 INFO:teuthology.orchestra.run.trial072.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 6.3 MB/s | 297 kB 00:00 2026-02-25T10:15:15.159 INFO:teuthology.orchestra.run.trial068.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 35 kB/s | 6.5 kB 00:00 2026-02-25T10:15:15.162 INFO:teuthology.orchestra.run.trial072.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 1.2 MB/s | 50 kB 00:00 2026-02-25T10:15:15.203 INFO:teuthology.orchestra.run.trial072.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 421 kB/s | 17 kB 00:00 2026-02-25T10:15:15.250 INFO:teuthology.orchestra.run.trial072.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 6.4 MB/s | 301 kB 00:00 2026-02-25T10:15:15.314 INFO:teuthology.orchestra.run.trial072.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 16 MB/s | 1.0 MB 00:00 2026-02-25T10:15:15.539 INFO:teuthology.orchestra.run.trial068.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 1.6 MB/s | 937 kB 00:00 2026-02-25T10:15:15.586 INFO:teuthology.orchestra.run.trial068.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 3.4 MB/s | 163 kB 00:00 2026-02-25T10:15:15.818 INFO:teuthology.orchestra.run.trial068.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 2.4 MB/s | 2.0 MB 00:00 2026-02-25T10:15:15.986 INFO:teuthology.orchestra.run.trial072.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 522 kB/s | 351 kB 00:00 2026-02-25T10:15:16.107 INFO:teuthology.orchestra.run.trial072.stdout:(17/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 32 MB/s | 51 MB 00:01 2026-02-25T10:15:16.108 INFO:teuthology.orchestra.run.trial072.stdout:(18/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 331 kB/s | 40 kB 00:00 2026-02-25T10:15:16.206 INFO:teuthology.orchestra.run.trial072.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 732 kB/s | 72 kB 00:00 2026-02-25T10:15:16.308 INFO:teuthology.orchestra.run.trial072.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 1.8 MB/s | 184 kB 00:00 2026-02-25T10:15:16.392 INFO:teuthology.orchestra.run.trial068.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 3.9 MB/s | 4.9 MB 00:01 2026-02-25T10:15:16.410 INFO:teuthology.orchestra.run.trial072.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 253 kB 00:00 2026-02-25T10:15:16.423 INFO:teuthology.orchestra.run.trial068.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 1.8 MB/s | 57 kB 00:00 2026-02-25T10:15:16.456 INFO:teuthology.orchestra.run.trial068.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 5.5 MB/s | 184 kB 00:00 2026-02-25T10:15:16.821 INFO:teuthology.orchestra.run.trial072.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 3.0 MB/s | 1.2 MB 00:00 2026-02-25T10:15:16.904 INFO:teuthology.orchestra.run.trial068.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 11 MB/s | 15 MB 00:01 2026-02-25T10:15:16.920 INFO:teuthology.orchestra.run.trial072.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2026-02-25T10:15:17.023 INFO:teuthology.orchestra.run.trial072.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 1.3 MB/s | 135 kB 00:00 2026-02-25T10:15:17.122 INFO:teuthology.orchestra.run.trial072.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 1.5 MB/s | 150 kB 00:00 2026-02-25T10:15:17.221 INFO:teuthology.orchestra.run.trial072.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 1.2 MB/s | 126 kB 00:00 2026-02-25T10:15:17.322 INFO:teuthology.orchestra.run.trial072.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 2.1 MB/s | 218 kB 00:00 2026-02-25T10:15:17.374 INFO:teuthology.orchestra.run.trial068.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 16 MB/s | 7.4 MB 00:00 2026-02-25T10:15:17.416 INFO:teuthology.orchestra.run.trial072.stdout:(28/96): libgfortran-11.5.0-14.el9.x86_64.rpm 608 kB/s | 794 kB 00:01 2026-02-25T10:15:17.422 INFO:teuthology.orchestra.run.trial072.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 1.8 MB/s | 182 kB 00:00 2026-02-25T10:15:17.423 INFO:teuthology.orchestra.run.trial068.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 459 kB/s | 22 kB 00:00 2026-02-25T10:15:17.480 INFO:teuthology.orchestra.run.trial068.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 5.1 MB/s | 297 kB 00:00 2026-02-25T10:15:17.509 INFO:teuthology.orchestra.run.trial072.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 339 kB/s | 30 kB 00:00 2026-02-25T10:15:17.534 INFO:teuthology.orchestra.run.trial068.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 931 kB/s | 50 kB 00:00 2026-02-25T10:15:17.591 INFO:teuthology.orchestra.run.trial068.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 306 kB/s | 17 kB 00:00 2026-02-25T10:15:17.592 INFO:teuthology.orchestra.run.trial072.stdout:(31/96): zip-3.0-35.el9.x86_64.rpm 1.5 MB/s | 266 kB 00:00 2026-02-25T10:15:17.661 INFO:teuthology.orchestra.run.trial068.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 4.2 MB/s | 301 kB 00:00 2026-02-25T10:15:17.676 INFO:teuthology.orchestra.run.trial072.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 178 kB/s | 15 kB 00:00 2026-02-25T10:15:17.756 INFO:teuthology.orchestra.run.trial072.stdout:(33/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 12 MB/s | 3.0 MB 00:00 2026-02-25T10:15:17.762 INFO:teuthology.orchestra.run.trial072.stdout:(34/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 2.8 MB/s | 246 kB 00:00 2026-02-25T10:15:17.840 INFO:teuthology.orchestra.run.trial072.stdout:(35/96): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2026-02-25T10:15:17.846 INFO:teuthology.orchestra.run.trial072.stdout:(36/96): openblas-0.3.29-1.el9.x86_64.rpm 503 kB/s | 42 kB 00:00 2026-02-25T10:15:18.015 INFO:teuthology.orchestra.run.trial072.stdout:(37/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 30 MB/s | 5.3 MB 00:00 2026-02-25T10:15:18.099 INFO:teuthology.orchestra.run.trial072.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 2.8 MB/s | 244 kB 00:00 2026-02-25T10:15:18.183 INFO:teuthology.orchestra.run.trial072.stdout:(39/96): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2026-02-25T10:15:18.185 INFO:teuthology.orchestra.run.trial072.stdout:(40/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 2.8 MB/s | 249 kB 00:00 2026-02-25T10:15:18.210 INFO:teuthology.orchestra.run.trial068.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 1.9 MB/s | 1.0 MB 00:00 2026-02-25T10:15:18.267 INFO:teuthology.orchestra.run.trial072.stdout:(41/96): python3-jmespath-1.0.1-1.el9.noarch.rp 566 kB/s | 48 kB 00:00 2026-02-25T10:15:18.272 INFO:teuthology.orchestra.run.trial072.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 2.0 MB/s | 177 kB 00:00 2026-02-25T10:15:18.347 INFO:teuthology.orchestra.run.trial068.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.5 MB/s | 351 kB 00:00 2026-02-25T10:15:18.356 INFO:teuthology.orchestra.run.trial072.stdout:(43/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2026-02-25T10:15:18.357 INFO:teuthology.orchestra.run.trial072.stdout:(44/96): python3-markupsafe-1.1.1-12.el9.x86_64 409 kB/s | 35 kB 00:00 2026-02-25T10:15:18.411 INFO:teuthology.orchestra.run.trial068.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 632 kB/s | 40 kB 00:00 2026-02-25T10:15:18.442 INFO:teuthology.orchestra.run.trial072.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.1 MB/s | 442 kB 00:00 2026-02-25T10:15:18.496 INFO:teuthology.orchestra.run.trial068.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 852 kB/s | 72 kB 00:00 2026-02-25T10:15:18.525 INFO:teuthology.orchestra.run.trial072.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 927 kB/s | 77 kB 00:00 2026-02-25T10:15:18.535 INFO:teuthology.orchestra.run.trial072.stdout:(47/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 34 MB/s | 6.1 MB 00:00 2026-02-25T10:15:18.615 INFO:teuthology.orchestra.run.trial072.stdout:(48/96): python3-protobuf-3.14.0-17.el9.noarch. 2.9 MB/s | 267 kB 00:00 2026-02-25T10:15:18.619 INFO:teuthology.orchestra.run.trial072.stdout:(49/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.8 MB/s | 157 kB 00:00 2026-02-25T10:15:18.659 INFO:teuthology.orchestra.run.trial068.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 4.8 MB/s | 794 kB 00:00 2026-02-25T10:15:18.699 INFO:teuthology.orchestra.run.trial072.stdout:(50/96): python3-pyasn1-modules-0.4.8-7.el9.noa 3.2 MB/s | 277 kB 00:00 2026-02-25T10:15:18.702 INFO:teuthology.orchestra.run.trial072.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 647 kB/s | 54 kB 00:00 2026-02-25T10:15:18.764 INFO:teuthology.orchestra.run.trial068.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 1.7 MB/s | 184 kB 00:00 2026-02-25T10:15:18.785 INFO:teuthology.orchestra.run.trial072.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 500 kB/s | 42 kB 00:00 2026-02-25T10:15:18.871 INFO:teuthology.orchestra.run.trial072.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.1 MB/s | 189 kB 00:00 2026-02-25T10:15:18.901 INFO:teuthology.orchestra.run.trial068.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.8 MB/s | 253 kB 00:00 2026-02-25T10:15:18.955 INFO:teuthology.orchestra.run.trial072.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-25T10:15:19.039 INFO:teuthology.orchestra.run.trial072.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 265 kB/s | 22 kB 00:00 2026-02-25T10:15:19.083 INFO:teuthology.orchestra.run.trial068.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 6.8 MB/s | 1.2 MB 00:00 2026-02-25T10:15:19.201 INFO:teuthology.orchestra.run.trial072.stdout:(56/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 38 MB/s | 19 MB 00:00 2026-02-25T10:15:19.203 INFO:teuthology.orchestra.run.trial072.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 5.1 MB/s | 862 kB 00:00 2026-02-25T10:15:19.204 INFO:teuthology.orchestra.run.trial068.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 887 kB/s | 106 kB 00:00 2026-02-25T10:15:19.311 INFO:teuthology.orchestra.run.trial068.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 1.2 MB/s | 135 kB 00:00 2026-02-25T10:15:19.356 INFO:teuthology.orchestra.run.trial072.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 127 kB/s | 19 kB 00:00 2026-02-25T10:15:19.434 INFO:teuthology.orchestra.run.trial068.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 1.2 MB/s | 150 kB 00:00 2026-02-25T10:15:19.529 INFO:teuthology.orchestra.run.trial072.stdout:(59/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.6 MB/s | 551 kB 00:00 2026-02-25T10:15:19.529 INFO:teuthology.orchestra.run.trial072.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 876 kB/s | 151 kB 00:00 2026-02-25T10:15:19.541 INFO:teuthology.orchestra.run.trial068.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 1.2 MB/s | 126 kB 00:00 2026-02-25T10:15:19.608 INFO:teuthology.orchestra.run.trial072.stdout:(61/96): python3-asyncssh-2.13.2-5.el9.noarch.r 6.8 MB/s | 548 kB 00:00 2026-02-25T10:15:19.608 INFO:teuthology.orchestra.run.trial072.stdout:(62/96): python3-autocommand-2.2.2-8.el9.noarch 373 kB/s | 29 kB 00:00 2026-02-25T10:15:19.642 INFO:teuthology.orchestra.run.trial072.stdout:(63/96): python3-backports-tarfile-1.2.0-1.el9. 1.7 MB/s | 60 kB 00:00 2026-02-25T10:15:19.645 INFO:teuthology.orchestra.run.trial072.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.2 MB/s | 43 kB 00:00 2026-02-25T10:15:19.674 INFO:teuthology.orchestra.run.trial072.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 996 kB/s | 32 kB 00:00 2026-02-25T10:15:19.676 INFO:teuthology.orchestra.run.trial072.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 451 kB/s | 14 kB 00:00 2026-02-25T10:15:19.703 INFO:teuthology.orchestra.run.trial068.stdout:(27/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 23 MB/s | 88 MB 00:03 2026-02-25T10:15:19.714 INFO:teuthology.orchestra.run.trial072.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.3 MB/s | 173 kB 00:00 2026-02-25T10:15:19.737 INFO:teuthology.orchestra.run.trial072.stdout:(68/96): python3-cherrypy-18.6.1-2.el9.noarch.r 5.7 MB/s | 358 kB 00:00 2026-02-25T10:15:19.746 INFO:teuthology.orchestra.run.trial068.stdout:(28/96): python3-urllib3-1.26.5-7.el9.noarch.rp 1.0 MB/s | 218 kB 00:00 2026-02-25T10:15:19.758 INFO:teuthology.orchestra.run.trial072.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 5.7 MB/s | 254 kB 00:00 2026-02-25T10:15:19.846 INFO:teuthology.orchestra.run.trial068.stdout:(29/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 15 MB/s | 51 MB 00:03 2026-02-25T10:15:19.869 INFO:teuthology.orchestra.run.trial068.stdout:(30/96): unzip-6.0-59.el9.x86_64.rpm 1.1 MB/s | 182 kB 00:00 2026-02-25T10:15:19.877 INFO:teuthology.orchestra.run.trial072.stdout:(70/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 15 MB/s | 2.0 MB 00:00 2026-02-25T10:15:19.883 INFO:teuthology.orchestra.run.trial072.stdout:(71/96): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2026-02-25T10:15:19.889 INFO:teuthology.orchestra.run.trial068.stdout:(31/96): zip-3.0-35.el9.x86_64.rpm 1.8 MB/s | 266 kB 00:00 2026-02-25T10:15:19.916 INFO:teuthology.orchestra.run.trial072.stdout:(72/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.6 MB/s | 139 kB 00:00 2026-02-25T10:15:19.919 INFO:teuthology.orchestra.run.trial072.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.5 MB/s | 56 kB 00:00 2026-02-25T10:15:19.947 INFO:teuthology.orchestra.run.trial072.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 344 kB/s | 11 kB 00:00 2026-02-25T10:15:19.950 INFO:teuthology.orchestra.run.trial072.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 564 kB/s | 18 kB 00:00 2026-02-25T10:15:19.979 INFO:teuthology.orchestra.run.trial072.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 724 kB/s | 23 kB 00:00 2026-02-25T10:15:19.982 INFO:teuthology.orchestra.run.trial072.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 617 kB/s | 20 kB 00:00 2026-02-25T10:15:20.011 INFO:teuthology.orchestra.run.trial072.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 620 kB/s | 19 kB 00:00 2026-02-25T10:15:20.014 INFO:teuthology.orchestra.run.trial072.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 830 kB/s | 26 kB 00:00 2026-02-25T10:15:20.096 INFO:teuthology.orchestra.run.trial072.stdout:(80/96): python3-kubernetes-26.1.0-3.el9.noarch 12 MB/s | 1.0 MB 00:00 2026-02-25T10:15:20.100 INFO:teuthology.orchestra.run.trial072.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 922 kB/s | 79 kB 00:00 2026-02-25T10:15:20.132 INFO:teuthology.orchestra.run.trial072.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.4 MB/s | 86 kB 00:00 2026-02-25T10:15:20.135 INFO:teuthology.orchestra.run.trial072.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.6 MB/s | 58 kB 00:00 2026-02-25T10:15:20.164 INFO:teuthology.orchestra.run.trial072.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 521 kB/s | 16 kB 00:00 2026-02-25T10:15:20.170 INFO:teuthology.orchestra.run.trial072.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.5 MB/s | 90 kB 00:00 2026-02-25T10:15:20.196 INFO:teuthology.orchestra.run.trial072.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 944 kB/s | 31 kB 00:00 2026-02-25T10:15:20.211 INFO:teuthology.orchestra.run.trial072.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 4.5 MB/s | 188 kB 00:00 2026-02-25T10:15:20.231 INFO:teuthology.orchestra.run.trial072.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2026-02-25T10:15:20.248 INFO:teuthology.orchestra.run.trial072.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.4 MB/s | 125 kB 00:00 2026-02-25T10:15:20.264 INFO:teuthology.orchestra.run.trial072.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.1 MB/s | 36 kB 00:00 2026-02-25T10:15:20.283 INFO:teuthology.orchestra.run.trial072.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 2.4 MB/s | 86 kB 00:00 2026-02-25T10:15:20.300 INFO:teuthology.orchestra.run.trial072.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.4 MB/s | 90 kB 00:00 2026-02-25T10:15:20.316 INFO:teuthology.orchestra.run.trial072.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.4 MB/s | 48 kB 00:00 2026-02-25T10:15:20.332 INFO:teuthology.orchestra.run.trial072.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 699 kB/s | 22 kB 00:00 2026-02-25T10:15:20.348 INFO:teuthology.orchestra.run.trial072.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 624 kB/s | 20 kB 00:00 2026-02-25T10:15:20.368 INFO:teuthology.orchestra.run.trial068.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 31 kB/s | 15 kB 00:00 2026-02-25T10:15:20.404 INFO:teuthology.orchestra.run.trial068.stdout:(33/96): flexiblas-3.0.4-9.el9.x86_64.rpm 53 kB/s | 30 kB 00:00 2026-02-25T10:15:20.926 INFO:teuthology.orchestra.run.trial068.stdout:(34/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 2.8 MB/s | 3.0 MB 00:01 2026-02-25T10:15:21.148 INFO:teuthology.orchestra.run.trial068.stdout:(35/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 316 kB/s | 246 kB 00:00 2026-02-25T10:15:21.149 INFO:teuthology.orchestra.run.trial068.stdout:(36/96): lua-5.4.4-4.el9.x86_64.rpm 253 kB/s | 188 kB 00:00 2026-02-25T10:15:21.505 INFO:teuthology.orchestra.run.trial068.stdout:(37/96): openblas-0.3.29-1.el9.x86_64.rpm 73 kB/s | 42 kB 00:00 2026-02-25T10:15:22.261 INFO:teuthology.orchestra.run.trial068.stdout:(38/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 4.8 MB/s | 5.3 MB 00:01 2026-02-25T10:15:22.272 INFO:teuthology.orchestra.run.trial068.stdout:(39/96): python3-babel-2.9.1-2.el9.noarch.rpm 5.3 MB/s | 6.0 MB 00:01 2026-02-25T10:15:22.273 INFO:teuthology.orchestra.run.trial068.stdout:(40/96): python3-devel-3.9.25-3.el9.x86_64.rpm 318 kB/s | 244 kB 00:00 2026-02-25T10:15:22.859 INFO:teuthology.orchestra.run.trial068.stdout:(41/96): python3-jmespath-1.0.1-1.el9.noarch.rp 81 kB/s | 48 kB 00:00 2026-02-25T10:15:23.002 INFO:teuthology.orchestra.run.trial068.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 242 kB/s | 177 kB 00:00 2026-02-25T10:15:23.027 INFO:teuthology.orchestra.run.trial068.stdout:(43/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 325 kB/s | 249 kB 00:00 2026-02-25T10:15:23.571 INFO:teuthology.orchestra.run.trial068.stdout:(44/96): python3-markupsafe-1.1.1-12.el9.x86_64 61 kB/s | 35 kB 00:00 2026-02-25T10:15:23.621 INFO:teuthology.orchestra.run.trial072.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 9.2 MB/s | 88 MB 00:09 2026-02-25T10:15:23.625 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:23.625 INFO:teuthology.orchestra.run.trial072.stdout:Total 22 MB/s | 226 MB 00:10 2026-02-25T10:15:23.805 INFO:teuthology.orchestra.run.trial068.stdout:(45/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 1.3 MB/s | 1.2 MB 00:00 2026-02-25T10:15:24.033 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:24.070 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:24.071 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:24.163 INFO:teuthology.orchestra.run.trial068.stdout:(46/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 5.4 MB/s | 6.1 MB 00:01 2026-02-25T10:15:24.399 INFO:teuthology.orchestra.run.trial068.stdout:(47/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 534 kB/s | 442 kB 00:00 2026-02-25T10:15:24.452 INFO:teuthology.orchestra.run.trial068.stdout:(48/96): python3-packaging-20.9-5.el9.noarch.rp 119 kB/s | 77 kB 00:00 2026-02-25T10:15:24.871 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:24.871 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:24.948 INFO:teuthology.orchestra.run.trial068.stdout:(49/96): python3-protobuf-3.14.0-17.el9.noarch. 341 kB/s | 267 kB 00:00 2026-02-25T10:15:25.125 INFO:teuthology.orchestra.run.trial068.stdout:(50/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 217 kB/s | 157 kB 00:00 2026-02-25T10:15:25.239 INFO:teuthology.orchestra.run.trial068.stdout:(51/96): python3-pyasn1-modules-0.4.8-7.el9.noa 352 kB/s | 277 kB 00:00 2026-02-25T10:15:25.546 INFO:teuthology.orchestra.run.trial068.stdout:(52/96): python3-requests-oauthlib-1.3.0-12.el9 90 kB/s | 54 kB 00:00 2026-02-25T10:15:25.699 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:25.701 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T10:15:25.705 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T10:15:25.727 INFO:teuthology.orchestra.run.trial072.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T10:15:25.761 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T10:15:25.767 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T10:15:25.769 INFO:teuthology.orchestra.run.trial072.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T10:15:25.774 INFO:teuthology.orchestra.run.trial072.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T10:15:25.797 INFO:teuthology.orchestra.run.trial072.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T10:15:25.810 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T10:15:25.812 INFO:teuthology.orchestra.run.trial068.stdout:(53/96): python3-toml-0.10.2-6.el9.noarch.rpm 73 kB/s | 42 kB 00:00 2026-02-25T10:15:25.813 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T10:15:25.819 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T10:15:25.821 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T10:15:25.832 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T10:15:25.838 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T10:15:25.862 INFO:teuthology.orchestra.run.trial072.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T10:15:25.865 INFO:teuthology.orchestra.run.trial072.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T10:15:25.915 INFO:teuthology.orchestra.run.trial072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T10:15:25.928 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T10:15:25.930 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T10:15:25.935 INFO:teuthology.orchestra.run.trial072.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T10:15:25.940 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T10:15:25.945 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T10:15:25.948 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T10:15:25.963 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T10:15:25.967 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T10:15:25.972 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T10:15:25.983 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T10:15:25.992 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T10:15:25.999 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T10:15:26.018 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T10:15:26.029 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T10:15:26.045 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T10:15:26.050 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T10:15:26.057 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T10:15:26.098 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T10:15:26.280 INFO:teuthology.orchestra.run.trial068.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 257 kB/s | 189 kB 00:00 2026-02-25T10:15:26.323 INFO:teuthology.orchestra.run.trial068.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 522 kB/s | 22 kB 00:00 2026-02-25T10:15:26.409 INFO:teuthology.orchestra.run.trial068.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 9.8 MB/s | 862 kB 00:00 2026-02-25T10:15:26.425 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T10:15:26.435 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T10:15:26.439 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T10:15:26.444 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T10:15:26.446 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T10:15:26.471 INFO:teuthology.orchestra.run.trial068.stdout:(57/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 137 kB/s | 90 kB 00:00 2026-02-25T10:15:26.474 INFO:teuthology.orchestra.run.trial072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T10:15:26.518 INFO:teuthology.orchestra.run.trial072.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T10:15:26.527 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T10:15:26.532 INFO:teuthology.orchestra.run.trial072.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T10:15:26.536 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T10:15:26.541 INFO:teuthology.orchestra.run.trial068.stdout:(58/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 14 MB/s | 19 MB 00:01 2026-02-25T10:15:26.542 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T10:15:26.545 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T10:15:26.552 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T10:15:26.556 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T10:15:26.584 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T10:15:26.595 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T10:15:26.601 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T10:15:26.608 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T10:15:26.627 INFO:teuthology.orchestra.run.trial068.stdout:(59/96): grpc-data-1.46.7-10.el9.noarch.rpm 124 kB/s | 19 kB 00:00 2026-02-25T10:15:26.642 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T10:15:26.750 INFO:teuthology.orchestra.run.trial068.stdout:(60/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.6 MB/s | 551 kB 00:00 2026-02-25T10:15:26.788 INFO:teuthology.orchestra.run.trial068.stdout:(61/96): luarocks-3.9.2-5.el9.noarch.rpm 617 kB/s | 151 kB 00:00 2026-02-25T10:15:26.789 INFO:teuthology.orchestra.run.trial068.stdout:(62/96): python3-autocommand-2.2.2-8.el9.noarch 753 kB/s | 29 kB 00:00 2026-02-25T10:15:26.824 INFO:teuthology.orchestra.run.trial068.stdout:(63/96): python3-asyncssh-2.13.2-5.el9.noarch.r 2.7 MB/s | 548 kB 00:00 2026-02-25T10:15:26.829 INFO:teuthology.orchestra.run.trial068.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.1 MB/s | 43 kB 00:00 2026-02-25T10:15:26.834 INFO:teuthology.orchestra.run.trial068.stdout:(65/96): python3-backports-tarfile-1.2.0-1.el9. 1.3 MB/s | 60 kB 00:00 2026-02-25T10:15:26.862 INFO:teuthology.orchestra.run.trial068.stdout:(66/96): python3-cachetools-4.2.4-1.el9.noarch. 844 kB/s | 32 kB 00:00 2026-02-25T10:15:26.866 INFO:teuthology.orchestra.run.trial068.stdout:(67/96): python3-certifi-2023.05.07-4.el9.noarc 374 kB/s | 14 kB 00:00 2026-02-25T10:15:26.869 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T10:15:26.892 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T10:15:26.896 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T10:15:26.901 INFO:teuthology.orchestra.run.trial068.stdout:(68/96): python3-cheroot-10.0.1-4.el9.noarch.rp 2.5 MB/s | 173 kB 00:00 2026-02-25T10:15:26.916 INFO:teuthology.orchestra.run.trial068.stdout:(69/96): python3-cherrypy-18.6.1-2.el9.noarch.r 6.5 MB/s | 358 kB 00:00 2026-02-25T10:15:26.917 INFO:teuthology.orchestra.run.trial068.stdout:(70/96): python3-google-auth-2.45.0-1.el9.noarc 4.9 MB/s | 254 kB 00:00 2026-02-25T10:15:26.955 INFO:teuthology.orchestra.run.trial072.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T10:15:26.956 INFO:teuthology.orchestra.run.trial072.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T10:15:26.960 INFO:teuthology.orchestra.run.trial068.stdout:(71/96): python3-grpcio-tools-1.46.7-10.el9.x86 3.2 MB/s | 144 kB 00:00 2026-02-25T10:15:26.964 INFO:teuthology.orchestra.run.trial068.stdout:(72/96): python3-influxdb-5.3.1-1.el9.noarch.rp 2.9 MB/s | 139 kB 00:00 2026-02-25T10:15:26.978 INFO:teuthology.orchestra.run.trial072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T10:15:26.999 INFO:teuthology.orchestra.run.trial068.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.4 MB/s | 56 kB 00:00 2026-02-25T10:15:27.003 INFO:teuthology.orchestra.run.trial068.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 271 kB/s | 11 kB 00:00 2026-02-25T10:15:27.037 INFO:teuthology.orchestra.run.trial068.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 466 kB/s | 18 kB 00:00 2026-02-25T10:15:27.043 INFO:teuthology.orchestra.run.trial068.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 588 kB/s | 23 kB 00:00 2026-02-25T10:15:27.073 INFO:teuthology.orchestra.run.trial068.stdout:(77/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2026-02-25T10:15:27.075 INFO:teuthology.orchestra.run.trial068.stdout:(78/96): python3-jaraco-context-6.0.1-3.el9.noa 518 kB/s | 20 kB 00:00 2026-02-25T10:15:27.081 INFO:teuthology.orchestra.run.trial068.stdout:(79/96): python3-jaraco-functools-3.5.0-2.el9.n 511 kB/s | 19 kB 00:00 2026-02-25T10:15:27.111 INFO:teuthology.orchestra.run.trial068.stdout:(80/96): python3-jaraco-text-4.0.0-2.el9.noarch 700 kB/s | 26 kB 00:00 2026-02-25T10:15:27.122 INFO:teuthology.orchestra.run.trial068.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2026-02-25T10:15:27.150 INFO:teuthology.orchestra.run.trial068.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.2 MB/s | 86 kB 00:00 2026-02-25T10:15:27.160 INFO:teuthology.orchestra.run.trial068.stdout:(83/96): python3-kubernetes-26.1.0-3.el9.noarch 12 MB/s | 1.0 MB 00:00 2026-02-25T10:15:27.162 INFO:teuthology.orchestra.run.trial068.stdout:(84/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2026-02-25T10:15:27.187 INFO:teuthology.orchestra.run.trial068.stdout:(85/96): python3-portend-3.1.0-2.el9.noarch.rpm 435 kB/s | 16 kB 00:00 2026-02-25T10:15:27.199 INFO:teuthology.orchestra.run.trial068.stdout:(86/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.3 MB/s | 90 kB 00:00 2026-02-25T10:15:27.200 INFO:teuthology.orchestra.run.trial068.stdout:(87/96): python3-repoze-lru-0.7-16.el9.noarch.r 810 kB/s | 31 kB 00:00 2026-02-25T10:15:27.237 INFO:teuthology.orchestra.run.trial068.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 1.5 MB/s | 59 kB 00:00 2026-02-25T10:15:27.243 INFO:teuthology.orchestra.run.trial068.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 2.8 MB/s | 125 kB 00:00 2026-02-25T10:15:27.265 INFO:teuthology.orchestra.run.trial068.stdout:(90/96): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2026-02-25T10:15:27.275 INFO:teuthology.orchestra.run.trial068.stdout:(91/96): python3-tempora-5.0.0-2.el9.noarch.rpm 947 kB/s | 36 kB 00:00 2026-02-25T10:15:27.284 INFO:teuthology.orchestra.run.trial068.stdout:(92/96): python3-typing-extensions-4.15.0-1.el9 2.1 MB/s | 86 kB 00:00 2026-02-25T10:15:27.303 INFO:teuthology.orchestra.run.trial068.stdout:(93/96): python3-websocket-client-1.2.3-2.el9.n 2.3 MB/s | 90 kB 00:00 2026-02-25T10:15:27.306 INFO:teuthology.orchestra.run.trial072.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T10:15:27.313 INFO:teuthology.orchestra.run.trial068.stdout:(94/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.3 MB/s | 48 kB 00:00 2026-02-25T10:15:27.322 INFO:teuthology.orchestra.run.trial068.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 582 kB/s | 22 kB 00:00 2026-02-25T10:15:27.342 INFO:teuthology.orchestra.run.trial068.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 511 kB/s | 20 kB 00:00 2026-02-25T10:15:27.348 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:27.349 INFO:teuthology.orchestra.run.trial068.stdout:Total 16 MB/s | 226 MB 00:13 2026-02-25T10:15:27.380 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T10:15:27.771 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:27.809 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:27.809 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:28.055 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T10:15:28.072 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T10:15:28.082 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T10:15:28.089 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T10:15:28.102 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T10:15:28.118 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T10:15:28.194 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T10:15:28.202 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T10:15:28.226 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T10:15:28.256 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T10:15:28.307 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T10:15:28.313 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T10:15:28.316 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T10:15:28.317 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T10:15:28.335 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T10:15:28.335 INFO:teuthology.orchestra.run.trial072.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T10:15:28.336 INFO:teuthology.orchestra.run.trial072.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T10:15:28.336 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:28.343 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T10:15:28.369 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T10:15:28.369 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T10:15:28.369 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:28.406 INFO:teuthology.orchestra.run.trial072.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T10:15:28.611 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:28.611 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:28.993 INFO:teuthology.orchestra.run.trial072.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T10:15:29.441 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:29.443 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T10:15:29.447 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T10:15:29.469 INFO:teuthology.orchestra.run.trial068.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T10:15:29.503 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T10:15:29.509 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T10:15:29.512 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T10:15:29.516 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T10:15:29.540 INFO:teuthology.orchestra.run.trial068.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T10:15:29.552 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T10:15:29.555 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T10:15:29.561 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T10:15:29.563 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T10:15:29.575 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T10:15:29.581 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T10:15:29.605 INFO:teuthology.orchestra.run.trial068.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T10:15:29.608 INFO:teuthology.orchestra.run.trial068.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T10:15:29.658 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T10:15:29.671 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T10:15:29.674 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T10:15:29.678 INFO:teuthology.orchestra.run.trial068.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T10:15:29.684 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T10:15:29.688 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T10:15:29.691 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T10:15:29.706 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T10:15:29.710 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T10:15:29.715 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T10:15:29.727 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T10:15:29.736 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T10:15:29.742 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T10:15:29.761 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T10:15:29.772 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T10:15:29.788 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T10:15:29.792 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T10:15:29.800 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T10:15:29.840 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T10:15:30.167 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T10:15:30.177 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T10:15:30.180 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T10:15:30.186 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T10:15:30.188 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T10:15:30.215 INFO:teuthology.orchestra.run.trial068.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T10:15:30.259 INFO:teuthology.orchestra.run.trial068.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T10:15:30.268 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T10:15:30.273 INFO:teuthology.orchestra.run.trial068.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T10:15:30.277 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T10:15:30.283 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T10:15:30.286 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T10:15:30.293 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T10:15:30.296 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T10:15:30.324 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T10:15:30.334 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T10:15:30.340 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T10:15:30.346 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T10:15:30.381 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T10:15:30.605 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T10:15:30.628 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T10:15:30.632 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T10:15:30.691 INFO:teuthology.orchestra.run.trial068.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T10:15:30.692 INFO:teuthology.orchestra.run.trial068.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T10:15:30.714 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T10:15:31.038 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T10:15:31.111 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T10:15:31.780 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T10:15:31.796 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T10:15:31.806 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T10:15:31.813 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T10:15:31.826 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T10:15:31.842 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T10:15:31.918 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T10:15:31.926 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T10:15:31.950 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T10:15:31.979 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T10:15:32.030 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T10:15:32.036 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T10:15:32.039 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T10:15:32.041 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T10:15:32.059 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T10:15:32.059 INFO:teuthology.orchestra.run.trial068.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T10:15:32.059 INFO:teuthology.orchestra.run.trial068.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T10:15:32.059 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:32.068 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T10:15:32.095 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T10:15:32.095 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T10:15:32.096 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:32.131 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T10:15:32.719 INFO:teuthology.orchestra.run.trial068.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T10:15:34.401 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T10:15:34.465 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T10:15:34.467 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T10:15:34.482 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T10:15:34.482 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:34.482 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T10:15:34.482 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:34.493 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T10:15:34.517 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T10:15:34.517 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:34.517 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T10:15:34.518 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T10:15:34.518 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T10:15:34.518 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:34.575 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T10:15:34.577 INFO:teuthology.orchestra.run.trial072.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T10:15:34.586 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T10:15:34.610 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T10:15:34.612 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T10:15:37.521 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T10:15:37.527 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T10:15:37.987 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T10:15:37.988 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T10:15:38.002 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T10:15:38.002 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T10:15:38.065 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T10:15:38.105 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T10:15:38.106 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T10:15:38.128 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T10:15:38.128 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:38.128 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T10:15:38.128 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T10:15:38.128 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T10:15:38.128 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:38.140 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T10:15:38.144 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T10:15:38.209 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T10:15:38.211 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T10:15:38.226 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T10:15:38.226 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:38.226 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T10:15:38.226 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:38.239 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T10:15:38.264 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T10:15:38.264 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:38.264 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T10:15:38.264 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T10:15:38.265 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T10:15:38.265 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:38.322 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T10:15:38.324 INFO:teuthology.orchestra.run.trial068.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T10:15:38.333 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T10:15:38.357 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T10:15:38.359 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T10:15:38.359 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T10:15:38.360 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T10:15:38.382 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T10:15:38.382 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:38.382 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T10:15:38.382 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T10:15:38.382 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T10:15:38.382 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:38.465 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T10:15:38.485 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T10:15:38.485 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:38.485 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T10:15:38.485 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T10:15:38.485 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T10:15:38.485 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:38.486 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T10:15:38.493 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T10:15:38.518 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T10:15:39.990 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T10:15:39.990 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T10:15:39.990 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T10:15:39.990 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T10:15:39.990 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T10:15:39.990 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T10:15:39.990 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T10:15:39.991 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T10:15:39.992 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T10:15:39.993 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T10:15:39.994 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T10:15:40.062 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T10:15:40.062 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:40.062 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:40.062 INFO:teuthology.orchestra.run.trial072.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.063 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T10:15:40.064 INFO:teuthology.orchestra.run.trial072.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T10:15:40.065 INFO:teuthology.orchestra.run.trial072.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T10:15:40.066 INFO:teuthology.orchestra.run.trial072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T10:15:40.067 INFO:teuthology.orchestra.run.trial072.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T10:15:40.068 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T10:15:40.069 INFO:teuthology.orchestra.run.trial072.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: zip-3.0-35.el9.x86_64 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:40.070 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:40.134 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-base 2026-02-25T10:15:40.457 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:59 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:40.524 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:40.543 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:40.543 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:40.543 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:40.566 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install cephadm 2026-02-25T10:15:40.887 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:59 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:40.953 INFO:teuthology.orchestra.run.trial072.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:40.972 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:40.973 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:40.973 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:40.995 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T10:15:41.259 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T10:15:41.266 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T10:15:41.313 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:00 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:41.399 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repo Size 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:Install 1 Package 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 149 k 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 455 k 2026-02-25T10:15:41.400 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:41.493 INFO:teuthology.orchestra.run.trial072.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 1.6 MB/s | 149 kB 00:00 2026-02-25T10:15:41.494 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:41.494 INFO:teuthology.orchestra.run.trial072.stdout:Total 1.6 MB/s | 149 kB 00:00 2026-02-25T10:15:41.495 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:41.498 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:41.498 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:41.530 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:41.530 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:41.585 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:41.586 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T10:15:41.610 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T10:15:41.610 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:41.610 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T10:15:41.610 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:41.721 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T10:15:41.722 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T10:15:41.734 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T10:15:41.735 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T10:15:41.796 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T10:15:41.836 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T10:15:41.838 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T10:15:41.859 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T10:15:41.859 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:41.859 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T10:15:41.859 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T10:15:41.859 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T10:15:41.859 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:41.870 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T10:15:41.931 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T10:15:41.931 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:41.931 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:41.931 INFO:teuthology.orchestra.run.trial072.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:41.931 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:41.932 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:41.958 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr 2026-02-25T10:15:42.085 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T10:15:42.087 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T10:15:42.108 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T10:15:42.109 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:42.109 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T10:15:42.109 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T10:15:42.109 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T10:15:42.109 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:42.193 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T10:15:42.215 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T10:15:42.215 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:42.215 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T10:15:42.215 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T10:15:42.215 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T10:15:42.215 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:42.216 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T10:15:42.223 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T10:15:42.249 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T10:15:42.282 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:01 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:42.349 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:42.368 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:42.369 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:42.369 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:42.391 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T10:15:42.709 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:01 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:42.775 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:42.794 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:42.794 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:42.794 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:42.816 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T10:15:43.135 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:02 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:43.201 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:43.220 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:43.221 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:43.221 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:43.243 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-rook 2026-02-25T10:15:43.566 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:02 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:43.633 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T10:15:43.648 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T10:15:43.649 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T10:15:43.650 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T10:15:43.651 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T10:15:43.652 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T10:15:43.653 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T10:15:43.654 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T10:15:43.655 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T10:15:43.656 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T10:15:43.656 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T10:15:43.656 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T10:15:43.656 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T10:15:43.656 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T10:15:43.656 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T10:15:43.656 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T10:15:43.657 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:43.657 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:43.657 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:43.674 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T10:15:43.716 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T10:15:43.716 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:43.717 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:43.717 INFO:teuthology.orchestra.run.trial068.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T10:15:43.717 INFO:teuthology.orchestra.run.trial068.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:43.717 INFO:teuthology.orchestra.run.trial068.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.717 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:43.717 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:43.717 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.718 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.718 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.718 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.718 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.718 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.718 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:43.718 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T10:15:43.719 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T10:15:43.720 INFO:teuthology.orchestra.run.trial068.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T10:15:43.721 INFO:teuthology.orchestra.run.trial068.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T10:15:43.722 INFO:teuthology.orchestra.run.trial068.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T10:15:43.723 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T10:15:43.724 INFO:teuthology.orchestra.run.trial068.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T10:15:43.725 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T10:15:43.726 INFO:teuthology.orchestra.run.trial068.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T10:15:43.727 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T10:15:43.728 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T10:15:43.728 INFO:teuthology.orchestra.run.trial068.stdout: zip-3.0-35.el9.x86_64 2026-02-25T10:15:43.728 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:43.728 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:43.784 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-base 2026-02-25T10:15:43.991 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:02 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:44.059 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:44.078 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:44.079 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:44.079 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:44.100 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-osd 2026-02-25T10:15:44.107 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:01 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:44.174 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:44.193 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:44.193 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:44.193 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:44.215 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install cephadm 2026-02-25T10:15:44.416 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:03 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:44.484 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:44.503 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:44.503 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:44.504 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:44.525 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-osd-classic 2026-02-25T10:15:44.533 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:01 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:44.600 INFO:teuthology.orchestra.run.trial068.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:44.619 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:44.619 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:44.620 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:44.640 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T10:15:44.843 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:03 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:44.910 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:44.929 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:44.929 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:44.929 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:44.950 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-fuse 2026-02-25T10:15:44.951 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:01 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:45.036 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:45.037 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-25T10:15:45.037 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:45.037 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 149 k 2026-02-25T10:15:45.037 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 455 k 2026-02-25T10:15:45.037 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:45.123 INFO:teuthology.orchestra.run.trial068.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 1.7 MB/s | 149 kB 00:00 2026-02-25T10:15:45.124 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:45.124 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.7 MB/s | 149 kB 00:00 2026-02-25T10:15:45.124 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:45.128 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:45.128 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:45.159 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:45.159 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:45.214 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:45.216 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T10:15:45.238 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T10:15:45.239 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:45.239 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T10:15:45.239 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:45.268 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:04 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:45.354 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:45.354 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:45.354 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repo Size 2026-02-25T10:15:45.354 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:45.354 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout:Install 1 Package 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 914 k 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 2.9 M 2026-02-25T10:15:45.355 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:45.475 INFO:teuthology.orchestra.run.trial072.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 7.4 MB/s | 914 kB 00:00 2026-02-25T10:15:45.475 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:45.475 INFO:teuthology.orchestra.run.trial072.stdout:Total 7.4 MB/s | 914 kB 00:00 2026-02-25T10:15:45.475 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:45.480 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:45.480 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:45.515 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:45.515 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:45.556 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T10:15:45.557 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:45.557 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:45.557 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:45.557 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:45.557 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:45.584 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr 2026-02-25T10:15:45.606 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:45.624 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:45.875 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:45.908 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:02 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:45.938 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:45.938 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:45.938 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:45.939 INFO:teuthology.orchestra.run.trial072.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:45.939 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:45.939 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:45.967 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-volume 2026-02-25T10:15:45.975 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:45.994 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:45.995 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:45.995 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:46.017 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T10:15:46.287 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:05 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:46.331 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:03 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:46.353 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:46.372 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:46.373 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:46.373 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:46.394 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install librados-devel 2026-02-25T10:15:46.397 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:46.416 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:46.417 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:46.417 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:46.437 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T10:15:46.708 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:05 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:46.759 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:03 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:46.793 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:46.794 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:46.794 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repo Size 2026-02-25T10:15:46.794 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:46.794 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:46.794 INFO:teuthology.orchestra.run.trial072.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout:Install 1 Package 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 124 k 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 434 k 2026-02-25T10:15:46.795 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:46.825 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:46.844 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:46.844 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:46.845 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:46.867 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-rook 2026-02-25T10:15:46.882 INFO:teuthology.orchestra.run.trial072.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 1.4 MB/s | 124 kB 00:00 2026-02-25T10:15:46.882 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:46.882 INFO:teuthology.orchestra.run.trial072.stdout:Total 1.4 MB/s | 124 kB 00:00 2026-02-25T10:15:46.883 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:46.885 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:46.885 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:46.916 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:46.916 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:46.971 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:46.989 INFO:teuthology.orchestra.run.trial072.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:47.079 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:47.141 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:47.141 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:47.141 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:47.141 INFO:teuthology.orchestra.run.trial072.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:47.141 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:47.141 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:47.169 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install libcephfs2 2026-02-25T10:15:47.178 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:04 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:47.243 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:47.261 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:47.262 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:47.262 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:47.283 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T10:15:47.488 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:06 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:47.553 INFO:teuthology.orchestra.run.trial072.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:47.572 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:47.572 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:47.573 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:47.594 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install libcephfs-devel 2026-02-25T10:15:47.597 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:04 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:47.662 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:47.681 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:47.682 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:47.682 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:47.703 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd 2026-02-25T10:15:47.911 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:06 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:47.998 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:47.998 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repo Size 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout:Install 1 Package 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:47.999 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 37 k 2026-02-25T10:15:48.000 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 171 k 2026-02-25T10:15:48.000 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:48.021 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:05 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:48.062 INFO:teuthology.orchestra.run.trial072.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 586 kB/s | 37 kB 00:00 2026-02-25T10:15:48.062 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:48.063 INFO:teuthology.orchestra.run.trial072.stdout:Total 578 kB/s | 37 kB 00:00 2026-02-25T10:15:48.063 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:48.064 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:48.065 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:48.067 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:48.067 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:48.087 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:48.089 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:48.106 INFO:teuthology.orchestra.run.trial072.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:48.106 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:48.107 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:48.107 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:48.127 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-classic 2026-02-25T10:15:48.166 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:48.229 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:48.229 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:48.229 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:48.230 INFO:teuthology.orchestra.run.trial072.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:48.230 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:48.230 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:48.256 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install librados2 2026-02-25T10:15:48.450 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:05 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:48.516 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:48.535 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:48.535 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:48.535 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:48.557 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-fuse 2026-02-25T10:15:48.571 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:07 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:48.638 INFO:teuthology.orchestra.run.trial072.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:48.655 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:48.656 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:48.656 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:48.678 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install librbd1 2026-02-25T10:15:48.878 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:05 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:48.964 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 914 k 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.9 M 2026-02-25T10:15:48.965 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:48.993 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:07 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:49.060 INFO:teuthology.orchestra.run.trial072.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:49.077 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:49.078 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:49.078 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:49.085 INFO:teuthology.orchestra.run.trial068.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 7.5 MB/s | 914 kB 00:00 2026-02-25T10:15:49.085 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:49.086 INFO:teuthology.orchestra.run.trial068.stdout:Total 7.4 MB/s | 914 kB 00:00 2026-02-25T10:15:49.086 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:49.090 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:49.090 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:49.100 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-rados 2026-02-25T10:15:49.124 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:49.124 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:49.216 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:49.234 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:49.418 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:08 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:49.480 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:49.484 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:49.503 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:49.504 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:49.504 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:49.525 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-rgw 2026-02-25T10:15:49.542 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:49.542 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:49.542 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:49.542 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:49.542 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:49.542 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:49.569 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-volume 2026-02-25T10:15:49.841 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:08 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:49.889 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:06 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:49.907 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:49.925 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:49.925 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:49.925 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:49.947 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-cephfs 2026-02-25T10:15:49.956 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:15:49.975 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:49.976 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:49.976 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:49.997 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados-devel 2026-02-25T10:15:50.264 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:09 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:50.314 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:07 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:50.330 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:50.348 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:50.348 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:50.348 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:50.370 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-rbd 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 124 k 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 434 k 2026-02-25T10:15:50.399 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:50.488 INFO:teuthology.orchestra.run.trial068.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 1.4 MB/s | 124 kB 00:00 2026-02-25T10:15:50.488 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:50.488 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.4 MB/s | 124 kB 00:00 2026-02-25T10:15:50.488 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:50.491 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:50.491 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:50.523 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:50.523 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:50.581 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:50.598 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:50.683 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:50.684 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:09 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:50.746 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:50.746 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:50.746 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:50.747 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:50.747 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:50.747 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:50.751 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:50.768 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:50.769 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:50.769 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:50.775 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs2 2026-02-25T10:15:50.791 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install rbd-fuse 2026-02-25T10:15:51.096 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:08 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:51.106 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:10 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:51.162 INFO:teuthology.orchestra.run.trial068.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:51.181 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:51.182 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:51.182 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:51.190 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout: Package Architecture Version Repository Size 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:51.191 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:51.192 INFO:teuthology.orchestra.run.trial072.stdout:Install 1 Package 2026-02-25T10:15:51.192 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:51.192 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 86 k 2026-02-25T10:15:51.192 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 236 k 2026-02-25T10:15:51.192 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:51.204 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs-devel 2026-02-25T10:15:51.265 INFO:teuthology.orchestra.run.trial072.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2026-02-25T10:15:51.265 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:51.265 INFO:teuthology.orchestra.run.trial072.stdout:Total 1.1 MB/s | 86 kB 00:00 2026-02-25T10:15:51.265 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:51.269 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:51.269 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:51.300 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:51.300 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:51.353 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:51.370 INFO:teuthology.orchestra.run.trial072.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:51.458 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:51.521 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:51.521 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:51.521 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:51.521 INFO:teuthology.orchestra.run.trial072.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:51.521 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:51.521 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:51.522 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:08 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:51.551 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install rbd-mirror 2026-02-25T10:15:51.606 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:51.606 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:51.606 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 37 k 2026-02-25T10:15:51.607 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 171 k 2026-02-25T10:15:51.608 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:51.827 INFO:teuthology.orchestra.run.trial068.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 169 kB/s | 37 kB 00:00 2026-02-25T10:15:51.827 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:51.827 INFO:teuthology.orchestra.run.trial068.stdout:Total 168 kB/s | 37 kB 00:00 2026-02-25T10:15:51.828 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:51.830 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:51.830 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:51.833 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:51.833 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:51.856 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:51.873 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:51.875 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:10 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:51.933 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repo Size 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:Install 1 Package 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:51.959 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 2.6 M 2026-02-25T10:15:51.960 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 11 M 2026-02-25T10:15:51.960 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:51.996 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:51.996 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:51.996 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:51.996 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:51.996 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:51.996 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:52.025 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados2 2026-02-25T10:15:52.267 INFO:teuthology.orchestra.run.trial072.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 8.5 MB/s | 2.6 MB 00:00 2026-02-25T10:15:52.267 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:52.267 INFO:teuthology.orchestra.run.trial072.stdout:Total 8.4 MB/s | 2.6 MB 00:00 2026-02-25T10:15:52.267 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:52.272 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:52.272 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:52.310 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:52.310 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:52.349 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:09 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:52.415 INFO:teuthology.orchestra.run.trial068.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:52.433 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:52.433 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:52.433 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:52.455 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librbd1 2026-02-25T10:15:52.473 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:52.475 INFO:teuthology.orchestra.run.trial072.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:52.498 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:52.498 INFO:teuthology.orchestra.run.trial072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:52.498 INFO:teuthology.orchestra.run.trial072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T10:15:52.498 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T10:15:52.498 INFO:teuthology.orchestra.run.trial072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T10:15:52.498 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:52.773 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:09 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:52.824 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:52.824 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:52.824 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:52.824 INFO:teuthology.orchestra.run.trial072.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:52.825 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:52.825 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:52.838 INFO:teuthology.orchestra.run.trial068.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:52.853 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install rbd-nbd 2026-02-25T10:15:52.856 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:52.857 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:52.857 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:52.878 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rados 2026-02-25T10:15:53.173 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:12 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:53.195 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:10 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:53.257 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout: Package Architecture Version Repository Size 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:53.258 INFO:teuthology.orchestra.run.trial072.stdout:Install 1 Package 2026-02-25T10:15:53.259 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:53.259 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 166 k 2026-02-25T10:15:53.259 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 496 k 2026-02-25T10:15:53.259 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:53.260 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:53.278 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:53.278 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:53.278 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:53.299 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rgw 2026-02-25T10:15:53.390 INFO:teuthology.orchestra.run.trial072.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 1.2 MB/s | 166 kB 00:00 2026-02-25T10:15:53.391 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:53.391 INFO:teuthology.orchestra.run.trial072.stdout:Total 1.2 MB/s | 166 kB 00:00 2026-02-25T10:15:53.391 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:15:53.395 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:15:53.395 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:15:53.427 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:15:53.427 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:15:53.484 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:15:53.501 INFO:teuthology.orchestra.run.trial072.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:53.589 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:53.617 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:10 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:53.653 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:53.653 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:53.653 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:15:53.654 INFO:teuthology.orchestra.run.trial072.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:53.654 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:53.654 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:53.682 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-osd-crimson 2026-02-25T10:15:53.683 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:53.701 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:53.701 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:53.701 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:53.722 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-cephfs 2026-02-25T10:15:53.999 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:12 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:54.043 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:11 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:54.065 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:54.083 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:54.084 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:15:54.084 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:15:54.107 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T10:15:54.110 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:54.128 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:54.129 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:54.129 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:54.150 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rbd 2026-02-25T10:15:54.426 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:01:13 ago on Wed 25 Feb 2026 10:14:41 AM UTC. 2026-02-25T10:15:54.464 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:11 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:54.510 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout: Package Arch Version Repo Size 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout:Installing: 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout:Installing weak dependencies: 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout:Transaction Summary 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout:================================================================================ 2026-02-25T10:15:54.511 INFO:teuthology.orchestra.run.trial072.stdout:Install 2 Packages 2026-02-25T10:15:54.512 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:15:54.512 INFO:teuthology.orchestra.run.trial072.stdout:Total download size: 445 M 2026-02-25T10:15:54.512 INFO:teuthology.orchestra.run.trial072.stdout:Installed size: 2.7 G 2026-02-25T10:15:54.512 INFO:teuthology.orchestra.run.trial072.stdout:Downloading Packages: 2026-02-25T10:15:54.530 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:54.548 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:54.548 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:54.548 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:54.569 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-fuse 2026-02-25T10:15:54.886 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:11 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:54.939 INFO:teuthology.orchestra.run.trial072.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 28 MB/s | 12 MB 00:00 2026-02-25T10:15:54.970 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:54.970 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:54.970 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-25T10:15:54.970 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 86 k 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 236 k 2026-02-25T10:15:54.971 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:55.142 INFO:teuthology.orchestra.run.trial068.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 505 kB/s | 86 kB 00:00 2026-02-25T10:15:55.142 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:55.142 INFO:teuthology.orchestra.run.trial068.stdout:Total 502 kB/s | 86 kB 00:00 2026-02-25T10:15:55.142 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:55.146 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:55.146 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:55.177 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:55.177 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:55.229 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:55.247 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:55.331 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:55.395 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:55.395 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:55.395 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:55.395 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:55.396 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:55.396 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:55.423 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-mirror 2026-02-25T10:15:55.747 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:12 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:55.833 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-25T10:15:55.834 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:55.834 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 2.6 M 2026-02-25T10:15:55.834 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 11 M 2026-02-25T10:15:55.834 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:56.629 INFO:teuthology.orchestra.run.trial068.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 3.3 MB/s | 2.6 MB 00:00 2026-02-25T10:15:56.629 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:56.629 INFO:teuthology.orchestra.run.trial068.stdout:Total 3.3 MB/s | 2.6 MB 00:00 2026-02-25T10:15:56.629 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:56.634 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:56.634 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:56.673 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:56.673 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:56.837 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:56.838 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:56.861 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:56.862 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T10:15:56.862 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T10:15:56.862 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T10:15:56.862 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T10:15:56.862 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:57.170 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:57.170 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:57.170 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:57.170 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:57.170 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:57.171 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:57.198 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-nbd 2026-02-25T10:15:57.522 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:14 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:57.607 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:57.607 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:57.607 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-25T10:15:57.607 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:57.607 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 166 k 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 496 k 2026-02-25T10:15:57.608 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:57.838 INFO:teuthology.orchestra.run.trial068.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 722 kB/s | 166 kB 00:00 2026-02-25T10:15:57.838 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:15:57.839 INFO:teuthology.orchestra.run.trial068.stdout:Total 719 kB/s | 166 kB 00:00 2026-02-25T10:15:57.839 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:15:57.843 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:15:57.843 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:15:57.874 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:15:57.874 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:15:57.931 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:15:57.948 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:58.040 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:58.105 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T10:15:58.105 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:58.105 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:15:58.105 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:15:58.106 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:58.106 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:58.134 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-crimson 2026-02-25T10:15:58.457 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:15 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:58.523 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:15:58.540 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:58.541 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:15:58.541 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:15:58.563 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T10:15:58.884 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:15 ago on Wed 25 Feb 2026 10:14:43 AM UTC. 2026-02-25T10:15:58.967 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:Installing weak dependencies: 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:Install 2 Packages 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:15:58.968 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 445 M 2026-02-25T10:15:58.969 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.7 G 2026-02-25T10:15:58.969 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-25T10:15:59.438 INFO:teuthology.orchestra.run.trial068.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 26 MB/s | 12 MB 00:00 2026-02-25T10:16:12.747 INFO:teuthology.orchestra.run.trial072.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 24 MB/s | 433 MB 00:18 2026-02-25T10:16:12.748 INFO:teuthology.orchestra.run.trial072.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:16:12.748 INFO:teuthology.orchestra.run.trial072.stdout:Total 24 MB/s | 445 MB 00:18 2026-02-25T10:16:12.748 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction check 2026-02-25T10:16:12.753 INFO:teuthology.orchestra.run.trial072.stdout:Transaction check succeeded. 2026-02-25T10:16:12.753 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction test 2026-02-25T10:16:14.126 INFO:teuthology.orchestra.run.trial072.stdout:Transaction test succeeded. 2026-02-25T10:16:14.126 INFO:teuthology.orchestra.run.trial072.stdout:Running transaction 2026-02-25T10:16:17.733 INFO:teuthology.orchestra.run.trial072.stdout: Preparing : 1/1 2026-02-25T10:16:18.457 INFO:teuthology.orchestra.run.trial068.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 22 MB/s | 433 MB 00:19 2026-02-25T10:16:18.458 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-25T10:16:18.458 INFO:teuthology.orchestra.run.trial068.stdout:Total 23 MB/s | 445 MB 00:19 2026-02-25T10:16:18.458 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-25T10:16:18.464 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-25T10:16:18.464 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-25T10:16:19.836 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-25T10:16:19.837 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-25T10:16:23.427 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-25T10:16:40.346 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T10:16:40.364 INFO:teuthology.orchestra.run.trial072.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T10:16:42.504 INFO:teuthology.orchestra.run.trial072.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T10:16:42.505 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T10:16:42.568 INFO:teuthology.orchestra.run.trial072.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T10:16:42.568 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:16:42.568 INFO:teuthology.orchestra.run.trial072.stdout:Installed: 2026-02-25T10:16:42.569 INFO:teuthology.orchestra.run.trial072.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:16:42.569 INFO:teuthology.orchestra.run.trial072.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:16:42.569 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:16:42.569 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:16:42.648 DEBUG:teuthology.parallel:result is None 2026-02-25T10:16:45.733 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T10:16:45.751 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T10:16:47.786 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T10:16:47.787 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T10:16:47.851 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T10:16:47.851 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:16:47.851 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-25T10:16:47.851 INFO:teuthology.orchestra.run.trial068.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:16:47.852 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T10:16:47.852 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:16:47.852 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:16:47.928 DEBUG:teuthology.parallel:result is None 2026-02-25T10:16:47.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:16:48.076 DEBUG:teuthology.orchestra.run.trial068:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T10:16:48.098 INFO:teuthology.orchestra.run.trial068.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T10:16:48.099 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T10:16:48.099 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T10:16:48.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:16:48.247 DEBUG:teuthology.orchestra.run.trial072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T10:16:48.269 INFO:teuthology.orchestra.run.trial072.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T10:16:48.269 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T10:16:48.269 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T10:16:48.273 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T10:16:48.273 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:16:48.273 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T10:16:48.300 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:16:48.300 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T10:16:48.334 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T10:16:48.334 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:16:48.335 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T10:16:48.364 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T10:16:48.428 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:16:48.429 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T10:16:48.455 DEBUG:teuthology.orchestra.run.trial072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T10:16:48.519 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T10:16:48.519 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:16:48.520 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T10:16:48.545 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T10:16:48.609 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:16:48.610 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T10:16:48.635 DEBUG:teuthology.orchestra.run.trial072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T10:16:48.699 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T10:16:48.699 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:16:48.699 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T10:16:48.725 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T10:16:48.790 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:16:48.790 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T10:16:48.815 DEBUG:teuthology.orchestra.run.trial072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T10:16:48.878 INFO:teuthology.run_tasks:Running task pexec... 2026-02-25T10:16:48.901 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-25T10:16:48.901 DEBUG:teuthology.orchestra.run.trial068:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T10:16:48.902 DEBUG:teuthology.orchestra.run.trial072:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T10:16:48.904 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:16:48.904 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T10:16:48.920 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:16:48.920 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T10:16:48.954 DEBUG:teuthology.parallel:result is None 2026-02-25T10:16:48.971 DEBUG:teuthology.parallel:result is None 2026-02-25T10:16:48.971 INFO:teuthology.run_tasks:Running task install... 2026-02-25T10:16:48.982 DEBUG:teuthology.task.install:project ceph 2026-02-25T10:16:48.983 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T10:16:48.983 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T10:16:48.983 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T10:16:48.992 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T10:16:48.992 INFO:teuthology.task.install:extra packages: [] 2026-02-25T10:16:48.992 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T10:16:48.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:16:48.994 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T10:16:48.995 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:16:49.184 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T10:16:49.184 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T10:16:49.191 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T10:16:49.191 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T10:16:49.387 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T10:16:49.388 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:16:49.388 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T10:16:49.402 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T10:16:49.402 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:16:49.402 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T10:16:49.413 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T10:16:49.414 DEBUG:teuthology.orchestra.run.trial068:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T10:16:49.426 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T10:16:49.427 DEBUG:teuthology.orchestra.run.trial072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T10:16:49.479 DEBUG:teuthology.orchestra.run.trial068:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T10:16:49.492 DEBUG:teuthology.orchestra.run.trial072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T10:16:49.543 DEBUG:teuthology.orchestra.run.trial068:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T10:16:49.558 DEBUG:teuthology.orchestra.run.trial072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T10:16:49.598 INFO:teuthology.orchestra.run.trial068.stdout:check_obsoletes = 1 2026-02-25T10:16:49.608 DEBUG:teuthology.orchestra.run.trial068:> sudo yum clean all 2026-02-25T10:16:49.613 INFO:teuthology.orchestra.run.trial072.stdout:check_obsoletes = 1 2026-02-25T10:16:49.622 DEBUG:teuthology.orchestra.run.trial072:> sudo yum clean all 2026-02-25T10:16:49.769 INFO:teuthology.orchestra.run.trial068.stdout:56 files removed 2026-02-25T10:16:49.781 INFO:teuthology.orchestra.run.trial072.stdout:56 files removed 2026-02-25T10:16:49.784 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-radosgw 2026-02-25T10:16:49.796 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-radosgw 2026-02-25T10:16:50.119 INFO:teuthology.orchestra.run.trial072.stdout:ceph packages for x86_64 568 kB/s | 94 kB 00:00 2026-02-25T10:16:50.120 INFO:teuthology.orchestra.run.trial068.stdout:ceph packages for x86_64 520 kB/s | 94 kB 00:00 2026-02-25T10:16:50.519 INFO:teuthology.orchestra.run.trial072.stdout:ceph noarch packages 66 kB/s | 24 kB 00:00 2026-02-25T10:16:50.520 INFO:teuthology.orchestra.run.trial068.stdout:ceph noarch packages 66 kB/s | 24 kB 00:00 2026-02-25T10:16:51.014 INFO:teuthology.orchestra.run.trial068.stdout:ceph source packages 4.4 kB/s | 2.1 kB 00:00 2026-02-25T10:16:51.015 INFO:teuthology.orchestra.run.trial072.stdout:ceph source packages 4.4 kB/s | 2.1 kB 00:00 2026-02-25T10:16:51.763 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-25T10:16:52.305 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.9 MB 00:01 2026-02-25T10:16:55.216 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 27 MB 00:02 2026-02-25T10:16:55.713 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 27 MB 00:02 2026-02-25T10:16:57.880 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - CRB 34 MB/s | 8.0 MB 00:00 2026-02-25T10:16:59.022 INFO:teuthology.orchestra.run.trial072.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 20 kB 00:00 2026-02-25T10:16:59.693 INFO:teuthology.orchestra.run.trial072.stdout:Extra Packages for Enterprise Linux 33 MB/s | 20 MB 00:00 2026-02-25T10:16:59.762 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 8.0 MB 00:01 2026-02-25T10:17:01.339 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 20 kB 00:00 2026-02-25T10:17:02.298 INFO:teuthology.orchestra.run.trial068.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-25T10:17:02.980 INFO:teuthology.orchestra.run.trial072.stdout:lab-extras 1.4 MB/s | 50 kB 00:00 2026-02-25T10:17:04.171 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:04.189 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:04.190 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:04.190 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:04.215 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-test 2026-02-25T10:17:04.539 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:04.605 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:04.623 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:04.623 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:04.625 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:04.646 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph 2026-02-25T10:17:04.964 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:05.031 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:05.048 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:05.049 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:05.049 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:05.071 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-base 2026-02-25T10:17:05.394 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:05.460 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:05.478 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:05.479 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:05.479 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:05.499 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install cephadm 2026-02-25T10:17:05.510 INFO:teuthology.orchestra.run.trial068.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T10:17:05.823 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:05.889 INFO:teuthology.orchestra.run.trial072.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:05.907 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:05.908 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:05.908 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:05.930 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T10:17:06.252 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:06.318 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:06.336 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:06.337 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:06.337 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:06.359 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr 2026-02-25T10:17:06.683 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:06.700 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:06.718 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:06.719 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:06.719 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:06.743 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-test 2026-02-25T10:17:06.749 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:06.768 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:06.768 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:06.768 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:06.790 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T10:17:07.065 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:07.116 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:07.130 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:07.148 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:07.149 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:07.149 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:07.170 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph 2026-02-25T10:17:07.182 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:07.200 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:07.200 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:07.201 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:07.223 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T10:17:07.488 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:07.543 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:07.554 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:07.572 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:07.573 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:07.573 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:07.594 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-base 2026-02-25T10:17:07.608 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:07.627 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:07.627 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:07.627 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:07.649 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-rook 2026-02-25T10:17:07.915 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:07.965 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:07.982 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:08.000 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:08.000 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:08.000 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:08.022 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install cephadm 2026-02-25T10:17:08.031 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:08.050 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:08.050 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:08.051 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:08.072 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T10:17:08.345 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:08.393 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:08.411 INFO:teuthology.orchestra.run.trial068.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:08.429 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:08.430 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:08.430 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:08.452 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T10:17:08.459 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:08.478 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:08.478 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:08.478 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:08.500 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-osd 2026-02-25T10:17:08.772 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:08.819 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:08.839 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:08.857 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:08.858 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:08.858 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:08.879 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr 2026-02-25T10:17:08.885 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:08.903 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:08.904 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:08.904 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:08.925 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-osd-classic 2026-02-25T10:17:09.197 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:09.251 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:09.263 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:09.281 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:09.282 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:09.282 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:09.303 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T10:17:09.318 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:09.336 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:09.336 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:09.336 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:09.358 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-fuse 2026-02-25T10:17:09.618 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:09.680 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:09.683 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:09.701 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:09.702 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:09.702 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:09.723 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T10:17:09.747 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:09.765 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:09.766 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:09.766 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:09.788 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install ceph-volume 2026-02-25T10:17:10.043 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:10.104 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:10.109 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:10.127 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:10.127 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:10.128 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:10.149 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-rook 2026-02-25T10:17:10.171 INFO:teuthology.orchestra.run.trial072.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:10.189 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:10.190 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:10.190 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:10.211 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install librados-devel 2026-02-25T10:17:10.467 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:10.534 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:10.534 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:10.552 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:10.553 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:10.553 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:10.574 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T10:17:10.600 INFO:teuthology.orchestra.run.trial072.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:10.618 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:10.619 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:10.619 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:10.641 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install libcephfs2 2026-02-25T10:17:10.898 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:10.962 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:10.964 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:10.982 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:10.983 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:10.983 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:11.004 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd 2026-02-25T10:17:11.028 INFO:teuthology.orchestra.run.trial072.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:11.046 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:11.047 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:11.047 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:11.069 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install libcephfs-devel 2026-02-25T10:17:11.324 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:11.387 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:11.390 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:11.408 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:11.409 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:11.409 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:11.431 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-classic 2026-02-25T10:17:11.453 INFO:teuthology.orchestra.run.trial072.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:11.471 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:11.471 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:11.472 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:11.493 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install librados2 2026-02-25T10:17:11.747 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:11.813 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:11.814 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:11.833 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:11.833 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:11.834 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:11.855 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-fuse 2026-02-25T10:17:11.879 INFO:teuthology.orchestra.run.trial072.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:11.897 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:11.898 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:11.898 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:11.920 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install librbd1 2026-02-25T10:17:12.173 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:12.240 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:12.247 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:12.258 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:12.258 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:12.259 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:12.281 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-volume 2026-02-25T10:17:12.313 INFO:teuthology.orchestra.run.trial072.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:12.331 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:12.331 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:12.331 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:12.354 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-rados 2026-02-25T10:17:12.602 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:12.668 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T10:17:12.672 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:12.686 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:12.687 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:12.687 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:12.709 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados-devel 2026-02-25T10:17:12.739 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:12.757 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:12.758 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:12.758 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:12.779 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-rgw 2026-02-25T10:17:13.024 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:13.090 INFO:teuthology.orchestra.run.trial068.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:13.102 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:13.107 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:13.108 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:13.108 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:13.129 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs2 2026-02-25T10:17:13.169 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:13.187 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:13.188 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:13.188 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:13.210 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-cephfs 2026-02-25T10:17:13.446 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:13.513 INFO:teuthology.orchestra.run.trial068.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:13.531 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:13.532 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:13.532 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:13.533 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:13.553 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs-devel 2026-02-25T10:17:13.599 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:13.617 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:13.618 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:13.618 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:13.640 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install python3-rbd 2026-02-25T10:17:13.876 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:13.942 INFO:teuthology.orchestra.run.trial068.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:13.960 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:13.961 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:13.961 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:13.963 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:13.983 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados2 2026-02-25T10:17:14.030 INFO:teuthology.orchestra.run.trial072.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:14.048 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:14.048 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:14.049 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:14.071 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install rbd-fuse 2026-02-25T10:17:14.302 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:14.368 INFO:teuthology.orchestra.run.trial068.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:14.386 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:14.387 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:14.387 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:14.390 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:14.408 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librbd1 2026-02-25T10:17:14.456 INFO:teuthology.orchestra.run.trial072.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:14.474 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:14.475 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:14.475 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:14.496 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install rbd-mirror 2026-02-25T10:17:14.730 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:14.797 INFO:teuthology.orchestra.run.trial068.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:14.814 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:14.815 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:14.815 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:14.817 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:14.837 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rados 2026-02-25T10:17:14.883 INFO:teuthology.orchestra.run.trial072.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:14.901 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:14.902 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:14.902 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:14.924 DEBUG:teuthology.orchestra.run.trial072:> sudo yum -y install rbd-nbd 2026-02-25T10:17:15.157 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:15.223 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:15.241 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:15.242 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:15.242 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:15.244 INFO:teuthology.orchestra.run.trial072.stdout:Last metadata expiration check: 0:00:13 ago on Wed 25 Feb 2026 10:17:02 AM UTC. 2026-02-25T10:17:15.264 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rgw 2026-02-25T10:17:15.310 INFO:teuthology.orchestra.run.trial072.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:15.328 INFO:teuthology.orchestra.run.trial072.stdout:Dependencies resolved. 2026-02-25T10:17:15.329 INFO:teuthology.orchestra.run.trial072.stdout:Nothing to do. 2026-02-25T10:17:15.329 INFO:teuthology.orchestra.run.trial072.stdout:Complete! 2026-02-25T10:17:15.354 DEBUG:teuthology.parallel:result is None 2026-02-25T10:17:15.583 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:15.649 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:15.667 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:15.667 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:15.668 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:15.689 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-cephfs 2026-02-25T10:17:16.013 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:16.080 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:16.098 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:16.098 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:16.099 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:16.120 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rbd 2026-02-25T10:17:16.442 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:16.509 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:16.526 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:16.527 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:16.527 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:16.549 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-fuse 2026-02-25T10:17:16.873 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:16.938 INFO:teuthology.orchestra.run.trial068.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:16.955 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:16.956 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:16.956 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:16.977 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-mirror 2026-02-25T10:17:17.300 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:17.366 INFO:teuthology.orchestra.run.trial068.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:17.384 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:17.385 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:17.385 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:17.407 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-nbd 2026-02-25T10:17:17.731 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 10:17:05 AM UTC. 2026-02-25T10:17:17.797 INFO:teuthology.orchestra.run.trial068.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T10:17:17.814 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-25T10:17:17.815 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-25T10:17:17.815 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-25T10:17:17.842 DEBUG:teuthology.parallel:result is None 2026-02-25T10:17:17.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:17:18.019 DEBUG:teuthology.orchestra.run.trial068:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T10:17:18.040 INFO:teuthology.orchestra.run.trial068.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T10:17:18.041 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T10:17:18.041 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T10:17:18.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T10:17:18.183 DEBUG:teuthology.orchestra.run.trial072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T10:17:18.205 INFO:teuthology.orchestra.run.trial072.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T10:17:18.205 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T10:17:18.205 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T10:17:18.208 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-25T10:17:18.208 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T10:17:18.287 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T10:17:18.287 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 777 /var/log/ceph 2026-02-25T10:17:18.289 DEBUG:teuthology.orchestra.run.trial072:> sudo chmod 777 /var/log/ceph 2026-02-25T10:17:18.313 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T10:17:18.313 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T10:17:18.354 DEBUG:teuthology.orchestra.run.trial072:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T10:17:18.377 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T10:17:18.377 DEBUG:teuthology.orchestra.run.trial068:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T10:17:18.417 DEBUG:teuthology.orchestra.run.trial072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T10:17:18.456 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T10:17:18.457 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T10:17:18.457 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115', 'branch': 'wip-logstoremanager', 'description': 'crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} thrashers/{default} thrashosds-health workloads/write_fadvise_dontneed}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '69115', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 869, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d', 'targets': {'trial068.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKgWGIuVULmLFMLXdWbNDTQ1G+/PHTTKZPaQZLiqMwp2X6mxIDJlXUENE9nBvitRmZsqzm6pT0Zf0MluMYLKkw=', 'trial072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbarXh6aldkIuv664JPlhZwLEshKDob0pFt7cPRnLfPqgP9wcCTl5wJ7Jrdx9xcz+eA44OmheAwhWNQIgUr7NY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'write': 100}, 'ops': 4000, 'write_fadvise_dontneed': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_07:55:35', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T10:17:18.458 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T10:17:18.480 DEBUG:teuthology.orchestra.run.trial072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T10:17:18.497 DEBUG:teuthology.orchestra.run.trial068:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T10:17:18.536 DEBUG:teuthology.orchestra.run.trial072:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T10:17:18.562 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:18.562 DEBUG:teuthology.orchestra.run.trial068:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T10:17:18.613 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T10:17:18.614 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_1 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-25 10:15:57.010029924 +0000 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:18.670 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:18.670 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T10:17:18.732 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-25T10:17:18.732 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-25T10:17:18.732 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000149022 s, 3.4 MB/s 2026-02-25T10:17:18.733 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T10:17:18.790 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_2 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1287 Links: 1 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-25 10:15:57.010029924 +0000 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-25 10:14:51.111962141 +0000 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-25 10:14:51.111962141 +0000 2026-02-25T10:17:18.846 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-25 10:14:51.111962141 +0000 2026-02-25T10:17:18.846 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T10:17:18.907 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-25T10:17:18.908 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-25T10:17:18.908 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.00014317 s, 3.6 MB/s 2026-02-25T10:17:18.909 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T10:17:18.965 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_3 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1297 Links: 1 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-25 10:15:57.011030025 +0000 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:19.021 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:19.021 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T10:17:19.082 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-25T10:17:19.083 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-25T10:17:19.083 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000177205 s, 2.9 MB/s 2026-02-25T10:17:19.084 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T10:17:19.140 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_4 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1306 Links: 1 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-25 10:15:57.011030025 +0000 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:19.197 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-25 10:14:51.112962101 +0000 2026-02-25T10:17:19.198 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T10:17:19.260 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-25T10:17:19.260 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-25T10:17:19.260 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000149472 s, 3.4 MB/s 2026-02-25T10:17:19.262 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T10:17:19.319 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T10:17:19.319 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:19.319 DEBUG:teuthology.orchestra.run.trial072:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T10:17:19.336 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T10:17:19.336 DEBUG:teuthology.orchestra.run.trial072:> stat /dev/vg_nvme/lv_1 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout:Device: 6h/6d Inode: 1311 Links: 1 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout:Access: 2026-02-25 10:15:52.634369981 +0000 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout:Modify: 2026-02-25 10:14:49.593518353 +0000 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout:Change: 2026-02-25 10:14:49.593518353 +0000 2026-02-25T10:17:19.392 INFO:teuthology.orchestra.run.trial072.stdout: Birth: 2026-02-25 10:14:49.593518353 +0000 2026-02-25T10:17:19.392 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T10:17:19.455 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records in 2026-02-25T10:17:19.455 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records out 2026-02-25T10:17:19.455 INFO:teuthology.orchestra.run.trial072.stderr:512 bytes copied, 0.000230913 s, 2.2 MB/s 2026-02-25T10:17:19.456 DEBUG:teuthology.orchestra.run.trial072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T10:17:19.513 DEBUG:teuthology.orchestra.run.trial072:> stat /dev/vg_nvme/lv_2 2026-02-25T10:17:19.569 INFO:teuthology.orchestra.run.trial072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T10:17:19.569 INFO:teuthology.orchestra.run.trial072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:19.569 INFO:teuthology.orchestra.run.trial072.stdout:Device: 6h/6d Inode: 1316 Links: 1 2026-02-25T10:17:19.569 INFO:teuthology.orchestra.run.trial072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:19.570 INFO:teuthology.orchestra.run.trial072.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:19.570 INFO:teuthology.orchestra.run.trial072.stdout:Access: 2026-02-25 10:15:52.634369981 +0000 2026-02-25T10:17:19.570 INFO:teuthology.orchestra.run.trial072.stdout:Modify: 2026-02-25 10:14:49.593518353 +0000 2026-02-25T10:17:19.570 INFO:teuthology.orchestra.run.trial072.stdout:Change: 2026-02-25 10:14:49.593518353 +0000 2026-02-25T10:17:19.570 INFO:teuthology.orchestra.run.trial072.stdout: Birth: 2026-02-25 10:14:49.593518353 +0000 2026-02-25T10:17:19.570 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T10:17:19.631 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records in 2026-02-25T10:17:19.632 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records out 2026-02-25T10:17:19.632 INFO:teuthology.orchestra.run.trial072.stderr:512 bytes copied, 0.000205876 s, 2.5 MB/s 2026-02-25T10:17:19.633 DEBUG:teuthology.orchestra.run.trial072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T10:17:19.690 DEBUG:teuthology.orchestra.run.trial072:> stat /dev/vg_nvme/lv_3 2026-02-25T10:17:19.746 INFO:teuthology.orchestra.run.trial072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T10:17:19.746 INFO:teuthology.orchestra.run.trial072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:19.746 INFO:teuthology.orchestra.run.trial072.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-25T10:17:19.746 INFO:teuthology.orchestra.run.trial072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:19.747 INFO:teuthology.orchestra.run.trial072.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:19.747 INFO:teuthology.orchestra.run.trial072.stdout:Access: 2026-02-25 10:15:52.635369979 +0000 2026-02-25T10:17:19.747 INFO:teuthology.orchestra.run.trial072.stdout:Modify: 2026-02-25 10:14:49.592518355 +0000 2026-02-25T10:17:19.747 INFO:teuthology.orchestra.run.trial072.stdout:Change: 2026-02-25 10:14:49.592518355 +0000 2026-02-25T10:17:19.747 INFO:teuthology.orchestra.run.trial072.stdout: Birth: 2026-02-25 10:14:49.592518355 +0000 2026-02-25T10:17:19.747 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T10:17:19.809 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records in 2026-02-25T10:17:19.809 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records out 2026-02-25T10:17:19.809 INFO:teuthology.orchestra.run.trial072.stderr:512 bytes copied, 0.000160471 s, 3.2 MB/s 2026-02-25T10:17:19.810 DEBUG:teuthology.orchestra.run.trial072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T10:17:19.867 DEBUG:teuthology.orchestra.run.trial072:> stat /dev/vg_nvme/lv_4 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout:Device: 6h/6d Inode: 1304 Links: 1 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout:Access: 2026-02-25 10:15:52.635369979 +0000 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout:Modify: 2026-02-25 10:14:49.592518355 +0000 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout:Change: 2026-02-25 10:14:49.592518355 +0000 2026-02-25T10:17:19.923 INFO:teuthology.orchestra.run.trial072.stdout: Birth: 2026-02-25 10:14:49.592518355 +0000 2026-02-25T10:17:19.923 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T10:17:19.986 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records in 2026-02-25T10:17:19.987 INFO:teuthology.orchestra.run.trial072.stderr:1+0 records out 2026-02-25T10:17:19.987 INFO:teuthology.orchestra.run.trial072.stderr:512 bytes copied, 0.000136206 s, 3.8 MB/s 2026-02-25T10:17:19.988 DEBUG:teuthology.orchestra.run.trial072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T10:17:20.045 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T10:17:20.046 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial068.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial072.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T10:17:20.046 INFO:tasks.ceph:Generating config... 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-25T10:17:20.047 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd delete sleep = 1 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd max backfills = 3 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] osd target transaction size = 5 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-25T10:17:20.048 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2026-02-25T10:17:20.048 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2026-02-25T10:17:20.048 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T10:17:20.048 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-25T10:17:20.048 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-25T10:17:20.048 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-25T10:17:20.049 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-25T10:17:20.049 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-25T10:17:20.049 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-25T10:17:20.049 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-25T10:17:20.049 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-25T10:17:20.049 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T10:17:20.049 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T10:17:20.091 INFO:teuthology.orchestra.run.trial068.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T10:17:20.094 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T10:17:20.177 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T10:17:20.241 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.68')] 2026-02-25T10:17:20.242 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.68', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-25T10:17:20.244 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:20.244 DEBUG:teuthology.orchestra.run.trial068:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T10:17:20.299 DEBUG:teuthology.orchestra.run.trial068:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.68 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:20.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:20.377+0000 7f6a67c9bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:20.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:20.377+0000 7f6a67c9bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:20.379 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:20.377+0000 7f6a67c9bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:monmaptool: generated fsid e959f599-67ec-4a01-98b7-0bdf9a6b1fe1 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:setting min_mon_release = tentacle 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:epoch 0 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:fsid e959f599-67ec-4a01-98b7-0bdf9a6b1fe1 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:last_changed 2026-02-25T10:17:20.378455+0000 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:created 2026-02-25T10:17:20.378455+0000 2026-02-25T10:17:20.381 INFO:teuthology.orchestra.run.trial068.stdout:min_mon_release 20 (tentacle) 2026-02-25T10:17:20.382 INFO:teuthology.orchestra.run.trial068.stdout:election_strategy: 1 2026-02-25T10:17:20.382 INFO:teuthology.orchestra.run.trial068.stdout:0: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.a 2026-02-25T10:17:20.382 INFO:teuthology.orchestra.run.trial068.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-25T10:17:20.382 DEBUG:teuthology.orchestra.run.trial068:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T10:17:20.438 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e959f599-67ec-4a01-98b7-0bdf9a6b1fe1... 2026-02-25T10:17:20.440 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T10:17:20.480 DEBUG:teuthology.orchestra.run.trial072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T10:17:20.519 INFO:teuthology.orchestra.run.trial068.stdout:[global] 2026-02-25T10:17:20.519 INFO:teuthology.orchestra.run.trial068.stdout: chdir = "" 2026-02-25T10:17:20.519 INFO:teuthology.orchestra.run.trial068.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: auth supported = cephx 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: filestore xattr use omap = true 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: mon clock drift allowed = 1.000 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: osd crush chooseleaf type = 0 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: auth debug = true 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: ms die on old message = true 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: ms die on bug = true 2026-02-25T10:17:20.520 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: mon pg warn max object skew = 0 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: osd pool default size = 2 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: mon osd allow primary affinity = true 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: mon osd allow pg remap = true 2026-02-25T10:17:20.521 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on legacy crush tunables = false 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on crush straw calc version zero = false 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on no sortbitwise = false 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on osd down out interval zero = false 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on too few osds = false 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: mon_allow_pool_size_one = true 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T10:17:20.522 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: osd default data pool replay window = 5 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: mon allow pool delete = true 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: mon cluster log file level = debug 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: debug asserts on shutdown = true 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: mon health detail to clog = false 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: mon host = 10.20.193.68 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: ms client mode = crc 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: ms cluster mode = crc 2026-02-25T10:17:20.523 INFO:teuthology.orchestra.run.trial068.stdout: ms mon client mode = crc 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: ms mon cluster mode = crc 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: ms mon service mode = crc 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: ms service mode = crc 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: osd_max_pg_log_entries = 2 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: osd_min_pg_log_entries = 1 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: osd_pg_log_trim_min = 0 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: osd_pool_default_min_size = 2 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: osd_pool_default_size = 3 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: fsid = e959f599-67ec-4a01-98b7-0bdf9a6b1fe1 2026-02-25T10:17:20.524 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout:[osd] 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd journal size = 100 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd scrub load threshold = 5.0 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd scrub max interval = 120 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock profile = high_recovery_ops 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock skip benchmark = true 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd recover clone overlap = true 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd recovery max chunk = 1048576 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.525 INFO:teuthology.orchestra.run.trial068.stdout: osd debug shutdown = true 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: osd debug op order = true 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: osd debug verify stray on activate = true 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: osd debug trim objects = true 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: osd open classes on start = true 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: osd debug pg log writeout = true 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: osd deep scrub update digest min age = 30 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: osd map max advance = 10 2026-02-25T10:17:20.526 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: journal zero on create = true 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: filestore ondisk finisher threads = 3 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: filestore apply finisher threads = 3 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: bdev debug aio = true 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: osd debug misdirected ops = true 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: debug monc = 20 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: crimson cpu num = 4 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: crimson osd obc lru size = 10 2026-02-25T10:17:20.527 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug osd = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore backref = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore cache = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore cleaner = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore device = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore epm = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore fixedkv tree = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore journal = 20 2026-02-25T10:17:20.528 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore lba = 20 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore odata = 20 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore omap = 20 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore onode = 20 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore t = 20 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: debug seastore tm = 20 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: osd blocked scrub grace period = 3600 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: osd debug reject backfill probability = 0.3 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: osd delete sleep = 1 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: osd max backfills = 3 2026-02-25T10:17:20.529 INFO:teuthology.orchestra.run.trial068.stdout: osd max markdown count = 1000 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: osd objectstore = seastore 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: osd scrub min interval = 60 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: osd shutdown pgref assert = True 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: osd snap trim sleep = 2 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: osd target transaction size = 5 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: seastore cachepin size pershard = 64M 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: seastore max concurrent transactions = 8 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout:[mgr] 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: debug mgr = 20 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: debug mon = 20 2026-02-25T10:17:20.530 INFO:teuthology.orchestra.run.trial068.stdout: debug auth = 20 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min pgs per osd = 4 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min bytes per osd = 10 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: mgr/telemetry/nag = false 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout:[mon] 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: debug mon = 20 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: debug paxos = 20 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: debug auth = 20 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: mon data avail warn = 5 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: mon mgr mkfs grace = 240 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min pgs per osd = 4 2026-02-25T10:17:20.531 INFO:teuthology.orchestra.run.trial068.stdout: mon osd reporter subtree level = osd 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: mon osd prime pg temp = true 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min bytes per osd = 10 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.532 INFO:teuthology.orchestra.run.trial068.stdout: # 1m isn't quite enough 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: mon_down_mkfs_grace = 2m 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_filestore_osds = false 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: mon min osdmap epochs = 50 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: mon osdmap full prune interval = 2 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: mon osdmap full prune min = 15 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: mon osdmap full prune txsize = 2 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: osd pool default crimson = True 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: paxos service trim min = 10 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout:[client] 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: rgw cache enabled = true 2026-02-25T10:17:20.533 INFO:teuthology.orchestra.run.trial068.stdout: rgw enable ops log = true 2026-02-25T10:17:20.534 INFO:teuthology.orchestra.run.trial068.stdout: rgw enable usage log = true 2026-02-25T10:17:20.534 INFO:teuthology.orchestra.run.trial068.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T10:17:20.534 INFO:teuthology.orchestra.run.trial068.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T10:17:20.534 INFO:teuthology.orchestra.run.trial068.stdout:[mon.a] 2026-02-25T10:17:20.536 INFO:teuthology.orchestra.run.trial072.stdout:[global] 2026-02-25T10:17:20.536 INFO:teuthology.orchestra.run.trial072.stdout: chdir = "" 2026-02-25T10:17:20.536 INFO:teuthology.orchestra.run.trial072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T10:17:20.536 INFO:teuthology.orchestra.run.trial072.stdout: auth supported = cephx 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: filestore xattr use omap = true 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: mon clock drift allowed = 1.000 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: osd crush chooseleaf type = 0 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: auth debug = true 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: ms die on old message = true 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: ms die on bug = true 2026-02-25T10:17:20.537 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: mon pg warn max object skew = 0 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: osd pool default size = 2 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: mon osd allow primary affinity = true 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: mon osd allow pg remap = true 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: mon warn on legacy crush tunables = false 2026-02-25T10:17:20.538 INFO:teuthology.orchestra.run.trial072.stdout: mon warn on crush straw calc version zero = false 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: mon warn on no sortbitwise = false 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: mon warn on osd down out interval zero = false 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: mon warn on too few osds = false 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: mon_allow_pool_size_one = true 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: osd default data pool replay window = 5 2026-02-25T10:17:20.539 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: mon allow pool delete = true 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: mon cluster log file level = debug 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: debug asserts on shutdown = true 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: mon health detail to clog = false 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: mon host = 10.20.193.68 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: ms client mode = crc 2026-02-25T10:17:20.540 INFO:teuthology.orchestra.run.trial072.stdout: ms cluster mode = crc 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: ms mon client mode = crc 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: ms mon cluster mode = crc 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: ms mon service mode = crc 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: ms service mode = crc 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: osd_max_pg_log_entries = 2 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: osd_min_pg_log_entries = 1 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: osd_pg_log_trim_min = 0 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: osd_pool_default_min_size = 2 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: osd_pool_default_size = 3 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: fsid = e959f599-67ec-4a01-98b7-0bdf9a6b1fe1 2026-02-25T10:17:20.541 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout:[osd] 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd journal size = 100 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd scrub load threshold = 5.0 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd scrub max interval = 120 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd mclock profile = high_recovery_ops 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd mclock skip benchmark = true 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd recover clone overlap = true 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd recovery max chunk = 1048576 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.542 INFO:teuthology.orchestra.run.trial072.stdout: osd debug shutdown = true 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: osd debug op order = true 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: osd debug verify stray on activate = true 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: osd debug trim objects = true 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: osd open classes on start = true 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: osd debug pg log writeout = true 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: osd deep scrub update digest min age = 30 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: osd map max advance = 10 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.543 INFO:teuthology.orchestra.run.trial072.stdout: journal zero on create = true 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: filestore ondisk finisher threads = 3 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: filestore apply finisher threads = 3 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: bdev debug aio = true 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: osd debug misdirected ops = true 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: debug monc = 20 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: crimson cpu num = 4 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: crimson osd obc lru size = 10 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: debug ms = 20 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: debug osd = 20 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore = 20 2026-02-25T10:17:20.544 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore backref = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore cache = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore cleaner = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore device = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore epm = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore fixedkv tree = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore journal = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore lba = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore odata = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore omap = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore onode = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore t = 20 2026-02-25T10:17:20.545 INFO:teuthology.orchestra.run.trial072.stdout: debug seastore tm = 20 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd blocked scrub grace period = 3600 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd debug reject backfill probability = 0.3 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd delete sleep = 1 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd max backfills = 3 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd max markdown count = 1000 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd objectstore = seastore 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd scrub min interval = 60 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd shutdown pgref assert = True 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd snap trim sleep = 2 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: osd target transaction size = 5 2026-02-25T10:17:20.546 INFO:teuthology.orchestra.run.trial072.stdout: seastore cachepin size pershard = 64M 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: seastore max concurrent transactions = 8 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout:[mgr] 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: debug ms = 1 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: debug mgr = 20 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: debug mon = 20 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: debug auth = 20 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: mon reweight min pgs per osd = 4 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: mon reweight min bytes per osd = 10 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: mgr/telemetry/nag = false 2026-02-25T10:17:20.547 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout:[mon] 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: debug ms = 1 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: debug mon = 20 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: debug paxos = 20 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: debug auth = 20 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: mon data avail warn = 5 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: mon mgr mkfs grace = 240 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: mon reweight min pgs per osd = 4 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: mon osd reporter subtree level = osd 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: mon osd prime pg temp = true 2026-02-25T10:17:20.548 INFO:teuthology.orchestra.run.trial072.stdout: mon reweight min bytes per osd = 10 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: # 1m isn't quite enough 2026-02-25T10:17:20.549 INFO:teuthology.orchestra.run.trial072.stdout: mon_down_mkfs_grace = 2m 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: mon_warn_on_filestore_osds = false 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: mon min osdmap epochs = 50 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: mon osdmap full prune interval = 2 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: mon osdmap full prune min = 15 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: mon osdmap full prune txsize = 2 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: osd pool default crimson = True 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: paxos service trim min = 10 2026-02-25T10:17:20.550 INFO:teuthology.orchestra.run.trial072.stdout: 2026-02-25T10:17:20.551 INFO:teuthology.orchestra.run.trial072.stdout:[client] 2026-02-25T10:17:20.551 INFO:teuthology.orchestra.run.trial072.stdout: rgw cache enabled = true 2026-02-25T10:17:20.551 INFO:teuthology.orchestra.run.trial072.stdout: rgw enable ops log = true 2026-02-25T10:17:20.551 INFO:teuthology.orchestra.run.trial072.stdout: rgw enable usage log = true 2026-02-25T10:17:20.551 INFO:teuthology.orchestra.run.trial072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T10:17:20.551 INFO:teuthology.orchestra.run.trial072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T10:17:20.551 INFO:teuthology.orchestra.run.trial072.stdout:[mon.a] 2026-02-25T10:17:20.551 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T10:17:20.552 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T10:17:20.625 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T10:17:20.626 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:20.626 DEBUG:teuthology.orchestra.run.trial068:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T10:17:20.642 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:20.643 DEBUG:teuthology.orchestra.run.trial068:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T10:17:20.699 INFO:tasks.ceph:Sending monmap to node ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:17:20.699 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:20.699 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T10:17:20.699 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T10:17:20.773 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:20.773 DEBUG:teuthology.orchestra.run.trial068:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:20.830 INFO:tasks.ceph:Sending monmap to node ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:17:20.830 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:20.831 DEBUG:teuthology.orchestra.run.trial072:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T10:17:20.831 DEBUG:teuthology.orchestra.run.trial072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T10:17:20.865 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:20.866 DEBUG:teuthology.orchestra.run.trial072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:20.921 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T10:17:20.922 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T10:17:20.922 DEBUG:teuthology.orchestra.run.trial072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T10:17:21.012 INFO:teuthology.orchestra.run.trial072.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T10:17:21.014 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T10:17:21.015 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T10:17:21.015 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T10:17:21.058 INFO:teuthology.orchestra.run.trial068.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T10:17:21.070 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-25T10:17:21.112 INFO:teuthology.orchestra.run.trial072.stdout:creating /etc/ceph/ceph.client.1.keyring 2026-02-25T10:17:21.124 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T10:17:21.124 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial068.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial072.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T10:17:21.124 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T10:17:21.148 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T10:17:21.148 INFO:tasks.ceph:role: osd.0 2026-02-25T10:17:21.148 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:17:21.148 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T10:17:21.213 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T10:17:21.214 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T10:17:21.784 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-25T10:17:21.787 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-25T10:17:21.787 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T10:17:21.855 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T10:17:21.924 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T10:17:21.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T10:17:21.989 INFO:tasks.ceph:role: osd.1 2026-02-25T10:17:21.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:17:21.989 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T10:17:22.055 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T10:17:22.055 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T10:17:22.055 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T10:17:22.055 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T10:17:22.056 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T10:17:22.056 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-25T10:17:22.056 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T10:17:22.056 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T10:17:22.056 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T10:17:22.056 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T10:17:22.616 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-25T10:17:22.617 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-25T10:17:22.618 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T10:17:22.686 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T10:17:22.753 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:22.902 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T10:17:22.958 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:22,957 seastar - Reactor backend: linux-aio 2026-02-25T10:17:22.968 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:22,967 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:22.968 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:22,967 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:22.970 INFO:teuthology.orchestra.run.trial068.stderr:==39667==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:23.027 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:23,026 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T10:17:23.028 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:23,026 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T10:17:23.028 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:23,026 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T10:17:23.028 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:23,026 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T10:17:23.028 INFO:teuthology.orchestra.run.trial068.stderr:DEBUG 2026-02-25 10:17:23,027 [shard 0:main] seastar - reactor::drain 2026-02-25T10:17:25.143 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:25.143 INFO:teuthology.orchestra.run.trial068.stderr:Suppressions used: 2026-02-25T10:17:25.143 INFO:teuthology.orchestra.run.trial068.stderr: count bytes template 2026-02-25T10:17:25.143 INFO:teuthology.orchestra.run.trial068.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:25.144 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:25.144 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-25T10:17:25.163 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T10:17:25.163 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T10:17:25.212 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T10:17:25.212 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T10:17:25.219 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:25,219 seastar - Reactor backend: linux-aio 2026-02-25T10:17:25.231 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:25,230 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:25.231 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:25,230 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:25.235 INFO:teuthology.orchestra.run.trial068.stderr:==39665==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:25.235 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:25,234 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T10:17:25.294 INFO:teuthology.orchestra.run.trial068.stderr:DEBUG 2026-02-25 10:17:25,294 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T10:17:25.376 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcc2811 0x10a28e0 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.376 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c62 /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe40f5d 0x9f19999 0x78b69db 0x7ba11e3 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.376 INFO:teuthology.orchestra.run.trial068.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbbf62f0b 0xffffffffbb5f9b78 0xffffffffbb5fa3e3 0xffffffffbb5fa864 0xffffffffbb5fc7f9 0xffffffffbb5fdd21 0xffffffffbb625a53 0xffffffffbb625c15 0xffffffffbb5cc8ce 0xffffffffbb5d4bab 0xffffffffbb5d500b 0xffffffffbb2910d2 0xffffffffbbf50451 0xffffffffbc000ba2 2026-02-25T10:17:25.376 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129749 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x3617fd7 0x9e33da7 0x9e33eaf 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.408 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xa7a7b7f 0x9f193cc 0x9f195b3 0x9f1977c 0x9f1980f 0x9f1a654 0x78b69db 0x7ba5cd9 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:25.451 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xea798 0xcf0980 0x9dc0e9f 0x9ef6e46 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.451 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x4c9fccb 0x9e303b9 0x9ef6683 0x9ef684b 0x9ef6976 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.476 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaf3 /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:25.516 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151be 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.517 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a68faa 0x7d56d00 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.517 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a695a0 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.639 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x9d747d8 0x9d74a5c 0x9d758a4 0x9eec474 0x9eec56d 0x7ba1825 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.639 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaef /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x318273f 0x9e21d5d 0x9e23b43 0x9f0751a 0x9f0916c 0x9f097c6 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.639 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x9ef64ee 0x9ef66e2 0x9ef684b 0x9ef6976 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.705 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15184 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.721 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15192 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:25.737 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151ab 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:25.738 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a1526e 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:27.936 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:27.936 INFO:teuthology.orchestra.run.trial068.stderr:Suppressions used: 2026-02-25T10:17:27.936 INFO:teuthology.orchestra.run.trial068.stderr: count bytes template 2026-02-25T10:17:27.936 INFO:teuthology.orchestra.run.trial068.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:27.936 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:27.936 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-25T10:17:27.971 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T10:17:27.997 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:28.148 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T10:17:28.204 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:28,203 seastar - Reactor backend: linux-aio 2026-02-25T10:17:28.222 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:28,221 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:28.222 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:28,221 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:28.224 INFO:teuthology.orchestra.run.trial068.stderr:==39732==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:28.281 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:28,280 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T10:17:28.281 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:28,280 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T10:17:28.281 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:28,280 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T10:17:28.281 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:28,280 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T10:17:28.281 INFO:teuthology.orchestra.run.trial068.stderr:DEBUG 2026-02-25 10:17:28,280 [shard 0:main] seastar - reactor::drain 2026-02-25T10:17:30.361 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:30.361 INFO:teuthology.orchestra.run.trial068.stderr:Suppressions used: 2026-02-25T10:17:30.361 INFO:teuthology.orchestra.run.trial068.stderr: count bytes template 2026-02-25T10:17:30.361 INFO:teuthology.orchestra.run.trial068.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:30.361 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:30.362 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-25T10:17:30.381 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T10:17:30.381 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T10:17:30.422 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T10:17:30.423 INFO:teuthology.orchestra.run.trial068.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T10:17:30.430 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:30,429 seastar - Reactor backend: linux-aio 2026-02-25T10:17:30.444 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:30,443 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:30.444 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:30,443 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:30.448 INFO:teuthology.orchestra.run.trial068.stderr:==39730==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:30.448 INFO:teuthology.orchestra.run.trial068.stderr:INFO 2026-02-25 10:17:30,447 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T10:17:30.509 INFO:teuthology.orchestra.run.trial068.stderr:DEBUG 2026-02-25 10:17:30,509 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T10:17:30.591 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x318273f 0x9f14aa0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.591 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16eaf /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x318273f 0x9e21d5d 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.591 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129859 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x9e21e70 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.624 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9eec45d 0x9eec56d 0x7b8fd92 0x7ba5769 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:30.624 INFO:teuthology.orchestra.run.trial068.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb41ff38 0xffffffffbb4203b0 0xffffffffbb29525e 0xffffffffbb5e7269 0xffffffffbb5ca7d5 0xffffffffbb5d4bab 0xffffffffbb5d500b 0xffffffffbb2910d2 0xffffffffbbf50451 0xffffffffbc000ba2 2026-02-25T10:17:30.666 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libstdc++.so.6+0x14960f /lib64/libstdc++.so.6+0x13b003 /lib64/libstdc++.so.6+0x13b3cb 0x7ba5957 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.666 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a27a4 0xf47938 0x9f044e9 0x9f04ad6 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7b90d7c 0x7ba55d5 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.691 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a689ca 0x7d56cf3 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:30.699 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e9a /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.733 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a68a25 0x7d56cf3 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.735 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a15260 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.861 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x157c7 0x361934d 0x9ef4143 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.861 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e30f9f 0x9e318ef 0x9f091d8 0x9f097c6 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.861 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed3 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x11380c8 0xf4792d 0x9e26106 0x9e26a45 0x9f12a15 0x9f13935 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.944 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69595 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:30.959 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a6865c 0x7d56cde 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.959 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6585 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:30.964 INFO:teuthology.orchestra.run.trial068.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:33.110 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:33.110 INFO:teuthology.orchestra.run.trial068.stderr:Suppressions used: 2026-02-25T10:17:33.111 INFO:teuthology.orchestra.run.trial068.stderr: count bytes template 2026-02-25T10:17:33.111 INFO:teuthology.orchestra.run.trial068.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:33.111 INFO:teuthology.orchestra.run.trial068.stderr:----------------------------------------------------- 2026-02-25T10:17:33.111 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-25T10:17:33.146 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T10:17:33.172 DEBUG:teuthology.orchestra.run.trial072:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T10:17:33.198 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T10:17:33.198 INFO:tasks.ceph:role: osd.2 2026-02-25T10:17:33.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:17:33.199 DEBUG:teuthology.orchestra.run.trial072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T10:17:33.263 INFO:teuthology.orchestra.run.trial072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T10:17:33.263 INFO:teuthology.orchestra.run.trial072.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T10:17:33.263 INFO:teuthology.orchestra.run.trial072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T10:17:33.264 INFO:teuthology.orchestra.run.trial072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T10:17:33.264 INFO:teuthology.orchestra.run.trial072.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T10:17:33.264 INFO:teuthology.orchestra.run.trial072.stdout: = sunit=0 swidth=0 blks 2026-02-25T10:17:33.264 INFO:teuthology.orchestra.run.trial072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T10:17:33.264 INFO:teuthology.orchestra.run.trial072.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T10:17:33.264 INFO:teuthology.orchestra.run.trial072.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T10:17:33.264 INFO:teuthology.orchestra.run.trial072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T10:17:33.831 INFO:teuthology.orchestra.run.trial072.stdout:Discarding blocks...Done. 2026-02-25T10:17:33.833 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial072.front.sepia.ceph.com -o noatime 2026-02-25T10:17:33.833 DEBUG:teuthology.orchestra.run.trial072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2026-02-25T10:17:33.901 DEBUG:teuthology.orchestra.run.trial072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T10:17:33.968 DEBUG:teuthology.orchestra.run.trial072:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-25T10:17:34.031 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T10:17:34.031 INFO:tasks.ceph:role: osd.3 2026-02-25T10:17:34.031 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:17:34.032 DEBUG:teuthology.orchestra.run.trial072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T10:17:34.097 INFO:teuthology.orchestra.run.trial072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T10:17:34.097 INFO:teuthology.orchestra.run.trial072.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T10:17:34.097 INFO:teuthology.orchestra.run.trial072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T10:17:34.097 INFO:teuthology.orchestra.run.trial072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T10:17:34.097 INFO:teuthology.orchestra.run.trial072.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T10:17:34.097 INFO:teuthology.orchestra.run.trial072.stdout: = sunit=0 swidth=0 blks 2026-02-25T10:17:34.097 INFO:teuthology.orchestra.run.trial072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T10:17:34.098 INFO:teuthology.orchestra.run.trial072.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T10:17:34.098 INFO:teuthology.orchestra.run.trial072.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T10:17:34.098 INFO:teuthology.orchestra.run.trial072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T10:17:34.655 INFO:teuthology.orchestra.run.trial072.stdout:Discarding blocks...Done. 2026-02-25T10:17:34.656 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial072.front.sepia.ceph.com -o noatime 2026-02-25T10:17:34.657 DEBUG:teuthology.orchestra.run.trial072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2026-02-25T10:17:34.725 DEBUG:teuthology.orchestra.run.trial072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-25T10:17:34.793 DEBUG:teuthology.orchestra.run.trial072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:34.946 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T10:17:35.003 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:35,003 seastar - Reactor backend: linux-aio 2026-02-25T10:17:35.016 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:35,016 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:35.017 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:35,016 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:35.019 INFO:teuthology.orchestra.run.trial072.stderr:==39270==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:35.075 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:35,073 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T10:17:35.075 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:35,073 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T10:17:35.075 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:35,074 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T10:17:35.075 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:35,074 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T10:17:35.075 INFO:teuthology.orchestra.run.trial072.stderr:DEBUG 2026-02-25 10:17:35,074 [shard 0:main] seastar - reactor::drain 2026-02-25T10:17:37.154 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:37.155 INFO:teuthology.orchestra.run.trial072.stderr:Suppressions used: 2026-02-25T10:17:37.155 INFO:teuthology.orchestra.run.trial072.stderr: count bytes template 2026-02-25T10:17:37.155 INFO:teuthology.orchestra.run.trial072.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:37.155 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:37.155 INFO:teuthology.orchestra.run.trial072.stderr: 2026-02-25T10:17:37.173 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T10:17:37.173 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T10:17:37.220 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T10:17:37.220 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T10:17:37.227 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:37,226 seastar - Reactor backend: linux-aio 2026-02-25T10:17:37.244 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:37,243 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:37.244 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:37,243 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:37.248 INFO:teuthology.orchestra.run.trial072.stderr:==39268==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:37.248 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:37,247 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T10:17:37.309 INFO:teuthology.orchestra.run.trial072.stderr:DEBUG 2026-02-25 10:17:37,308 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T10:17:37.390 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x755a777 0x7395f10 0x73e6c79 0x731e88c 0x78b693f 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.390 INFO:teuthology.orchestra.run.trial072.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9f491052 0xffffffffa0150451 0xffffffffa0200ba2 2026-02-25T10:17:37.391 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xb7ac6c 0x10a27ac 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e22e1c 0x9e2338a 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.391 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xe29e31 0x1138071 0xf4792d 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e328d9 0x9e32ceb 0x9f0353e 0x9f03972 0x9f03d05 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.423 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9eec45d 0x9eec56d 0x7ba6440 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:37.423 INFO:teuthology.orchestra.run.trial072.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9f7f7c9f 0xffffffff9f7fc74a 0xffffffff9f7fdd21 0xffffffff9f825a53 0xffffffff9f825c15 0xffffffff9f7ca888 0xffffffff9f7d4bab 0xffffffff9f7d500b 0xffffffff9f4910d2 0xffffffffa0150451 0xffffffffa0200ba2 2026-02-25T10:17:37.490 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ebc /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:37.530 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x7d56cde 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.530 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.531 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.661 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xef924 0x3185724 0x31858fe 0x3185bb5 0x3185b7a 0x3185caa 0x9f169a9 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.662 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe40f5d 0x9f16c08 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.662 INFO:teuthology.orchestra.run.trial072.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9f78bd47 0xffffffff9f7ca7ef 0xffffffff9f7d4bab 0xffffffff9f7d500b 0xffffffff9f4910d2 0xffffffffa0150451 0xffffffffa0200ba2 2026-02-25T10:17:37.662 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10a2823 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.744 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a694c9 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:37.759 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69509 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.760 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521e 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:37.760 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:39.957 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:39.958 INFO:teuthology.orchestra.run.trial072.stderr:Suppressions used: 2026-02-25T10:17:39.958 INFO:teuthology.orchestra.run.trial072.stderr: count bytes template 2026-02-25T10:17:39.958 INFO:teuthology.orchestra.run.trial072.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:39.958 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:39.958 INFO:teuthology.orchestra.run.trial072.stderr: 2026-02-25T10:17:39.992 DEBUG:teuthology.orchestra.run.trial072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T10:17:40.018 DEBUG:teuthology.orchestra.run.trial072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:40.171 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T10:17:40.223 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:40,222 seastar - Reactor backend: linux-aio 2026-02-25T10:17:40.233 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:40,232 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:40.233 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:40,232 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:40.235 INFO:teuthology.orchestra.run.trial072.stderr:==39334==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:40.292 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:40,291 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T10:17:40.292 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:40,291 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T10:17:40.292 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:40,291 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T10:17:40.292 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:40,291 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T10:17:40.292 INFO:teuthology.orchestra.run.trial072.stderr:DEBUG 2026-02-25 10:17:40,291 [shard 0:main] seastar - reactor::drain 2026-02-25T10:17:42.384 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:42.385 INFO:teuthology.orchestra.run.trial072.stderr:Suppressions used: 2026-02-25T10:17:42.385 INFO:teuthology.orchestra.run.trial072.stderr: count bytes template 2026-02-25T10:17:42.385 INFO:teuthology.orchestra.run.trial072.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:42.385 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:42.385 INFO:teuthology.orchestra.run.trial072.stderr: 2026-02-25T10:17:42.404 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T10:17:42.404 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T10:17:42.452 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T10:17:42.452 INFO:teuthology.orchestra.run.trial072.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T10:17:42.458 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:42,458 seastar - Reactor backend: linux-aio 2026-02-25T10:17:42.470 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:42,470 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:42.471 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:42,470 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T10:17:42.475 INFO:teuthology.orchestra.run.trial072.stderr:==39332==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T10:17:42.476 INFO:teuthology.orchestra.run.trial072.stderr:INFO 2026-02-25 10:17:42,475 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T10:17:42.536 INFO:teuthology.orchestra.run.trial072.stderr:DEBUG 2026-02-25 10:17:42,535 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T10:17:42.617 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e31ed4 0x9e323ab 0x9f0f898 0x9f0fe6a 0x9f1a337 0x78b69db 0x7ba11e3 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.617 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x157c7 0x10a2823 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.617 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb044e /lib64/libasan.so.8+0xb1437 0xb7ac87 0x10a27ac 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.650 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 66 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15e5f 0x9e21a6a 0x9e21e1c 0x9e23b43 0x9f09a9d 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:42.692 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed3 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x3617fd7 0x9e33da7 0x9e33eaf 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1568c 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.693 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed3 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x157c7 0x35fc095 0x35fc215 0x3618092 0x9e33da7 0x9e33e8a 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7b90d7c 0x7ba55d5 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.728 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69624 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:42.758 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1294d6 /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.758 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a694da 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.761 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.890 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x9dde226 0x9e1af39 0x9eec204 0x9f18680 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.890 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a295d 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.890 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc08 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.940 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a68680 0x7d56cde 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T10:17:42.987 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a695a0 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.987 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:42.988 INFO:teuthology.orchestra.run.trial072.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a6955f 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T10:17:45.180 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:45.180 INFO:teuthology.orchestra.run.trial072.stderr:Suppressions used: 2026-02-25T10:17:45.180 INFO:teuthology.orchestra.run.trial072.stderr: count bytes template 2026-02-25T10:17:45.181 INFO:teuthology.orchestra.run.trial072.stderr: 1 8 MallocExtension::Register 2026-02-25T10:17:45.181 INFO:teuthology.orchestra.run.trial072.stderr:----------------------------------------------------- 2026-02-25T10:17:45.181 INFO:teuthology.orchestra.run.trial072.stderr: 2026-02-25T10:17:45.216 DEBUG:teuthology.orchestra.run.trial072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-25T10:17:45.242 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T10:17:45.243 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:45.243 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T10:17:45.269 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:45.269 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T10:17:45.333 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:45.333 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T10:17:45.358 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:45.359 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T10:17:45.422 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:45.422 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-25T10:17:45.486 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:45.486 DEBUG:teuthology.orchestra.run.trial068:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T10:17:45.504 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:45.505 DEBUG:teuthology.orchestra.run.trial072:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2026-02-25T10:17:45.542 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T10:17:45.542 DEBUG:teuthology.orchestra.run.trial068:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T10:17:45.567 INFO:teuthology.orchestra.run.trial068.stdout:[osd.0] 2026-02-25T10:17:45.567 INFO:teuthology.orchestra.run.trial068.stdout: key = AQA1zJ5pUj2/ERAAimXmO8tB6VvVCW0oiCIJ4Q== 2026-02-25T10:17:45.567 INFO:teuthology.orchestra.run.trial068.stdout:[osd.1] 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout: key = AQA6zJ5pO3eQHhAAE0HO3ZgfS9RlVVCOVr94pg== 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout:[mgr.x] 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout: key = AQAxzJ5pERmuABAAzs55FxtuG6Gtd9vI2ZHP1A== 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout:[osd.2] 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBBzJ5pmMScEhAAxoF+0NIonk5kpJlU8Op65Q== 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout:[osd.3] 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBGzJ5pPEohIBAAr2b/dDz14R1/AQEow6kELg== 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout:[client.0] 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout: key = AQAxzJ5pS9l2AxAA3uo1uWbQNKTk1DBRyheSgg== 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout:[client.1] 2026-02-25T10:17:45.568 INFO:teuthology.orchestra.run.trial068.stdout: key = AQAxzJ5pmaCvBhAATD5pfceeuzBV7S2thE8iBw== 2026-02-25T10:17:45.570 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T10:17:45.658 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T10:17:45.708 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T10:17:45.757 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T10:17:45.805 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T10:17:45.852 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T10:17:45.901 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T10:17:45.950 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T10:17:45.951 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T10:17:45.974 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T10:17:46.060 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.058+0000 7facd0484c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.060 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.058+0000 7facd0484c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.072 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T10:17:46.097 DEBUG:teuthology.orchestra.run.trial068:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T10:17:46.152 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T10:17:46.153 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T10:17:46.153 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T10:17:46.196 INFO:tasks.ceph.mon.a:Started 2026-02-25T10:17:46.196 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T10:17:46.196 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T10:17:46.196 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T10:17:46.199 INFO:tasks.ceph.mgr.x:Started 2026-02-25T10:17:46.199 DEBUG:tasks.ceph:set 0 configs 2026-02-25T10:17:46.199 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph config dump 2026-02-25T10:17:46.255 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-25T10:17:46.253+0000 7f40b9887c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.255 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-25T10:17:46.253+0000 7f40b9887c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.255 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-25T10:17:46.253+0000 7f40b9887c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.260 INFO:tasks.ceph.mgr.x.trial072.stderr:2026-02-25T10:17:46.258+0000 7f459922efc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.276 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.272+0000 7f909f98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.279 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.278+0000 7f909f98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.284 INFO:tasks.ceph.mgr.x.trial072.stderr:2026-02-25T10:17:46.283+0000 7f459922efc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.284 INFO:tasks.ceph.mgr.x.trial072.stderr:2026-02-25T10:17:46.283+0000 7f459922efc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.313 INFO:teuthology.orchestra.run.trial068.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T10:17:46.321 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T10:17:46.321 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T10:17:46.396 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.391+0000 7fe40aa11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.397+0000 7fe40aa11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.433 INFO:teuthology.orchestra.run.trial068.stderr:adjusted tunables profile to default 2026-02-25T10:17:46.441 INFO:tasks.ceph:check_enable_crimson: True 2026-02-25T10:17:46.441 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-25T10:17:46.441 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-25T10:17:46.513 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.509+0000 7f263a79c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:46.514+0000 7f263a79c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:46.557 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-25T10:17:46.555+0000 7f40b0ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-25T10:17:46.557 INFO:tasks.ceph.mon.a.trial068.stderr:Please be aware that this feature is experimental, untested, 2026-02-25T10:17:46.557 INFO:tasks.ceph.mon.a.trial068.stderr:unsupported, and may result in data corruption, data loss, 2026-02-25T10:17:46.557 INFO:tasks.ceph.mon.a.trial068.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-25T10:17:46.557 INFO:tasks.ceph.mon.a.trial068.stderr:feature with important data. 2026-02-25T10:17:46.557 INFO:tasks.ceph.mon.a.trial068.stderr: 2026-02-25T10:17:47.131 INFO:tasks.ceph.mgr.x.trial072.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-25T10:17:47.131 INFO:tasks.ceph.mgr.x.trial072.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-25T10:17:47.131 INFO:tasks.ceph.mgr.x.trial072.stderr: from numpy import show_config as show_numpy_config 2026-02-25T10:17:47.445 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T10:17:47.445 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:47.445 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T10:17:47.470 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-25T10:17:47.471 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T10:17:47.535 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:47.536 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T10:17:47.559 DEBUG:teuthology.orchestra.run.trial072:> set -ex 2026-02-25T10:17:47.559 DEBUG:teuthology.orchestra.run.trial072:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-25T10:17:47.622 DEBUG:teuthology.orchestra.run.trial072:> sudo ceph --cluster ceph osd new 2ba36d3c-8fb4-4109-b054-75e9bf8b1a94 0 2026-02-25T10:17:47.738 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:47.733+0000 7fefcde5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:47.740 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:47.739+0000 7fefcde5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:47.778 INFO:teuthology.orchestra.run.trial072.stdout:0 2026-02-25T10:17:47.787 DEBUG:teuthology.orchestra.run.trial072:> sudo ceph --cluster ceph osd new 3dc31ea2-f09f-4175-b190-bec35c045b2c 1 2026-02-25T10:17:47.860 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:47.856+0000 7febd95dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:47.862 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:47.861+0000 7febd95dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:47.900 INFO:teuthology.orchestra.run.trial072.stdout:1 2026-02-25T10:17:47.909 DEBUG:teuthology.orchestra.run.trial072:> sudo ceph --cluster ceph osd new 699685d4-5d4f-4310-b2e5-4ce775e56238 2 2026-02-25T10:17:47.984 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:47.979+0000 7f8588e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:47.986 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:47.985+0000 7f8588e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:48.024 INFO:teuthology.orchestra.run.trial072.stdout:2 2026-02-25T10:17:48.032 DEBUG:teuthology.orchestra.run.trial072:> sudo ceph --cluster ceph osd new 459dac3b-8203-4a0d-8568-da567eec7c29 3 2026-02-25T10:17:48.106 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:48.102+0000 7f17ec5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:48.108 INFO:teuthology.orchestra.run.trial072.stderr:2026-02-25T10:17:48.107+0000 7f17ec5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:48.146 INFO:teuthology.orchestra.run.trial072.stdout:3 2026-02-25T10:17:48.154 DEBUG:teuthology.orchestra.run.trial068:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-25T10:17:48.184 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T10:17:48.185 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T10:17:48.227 INFO:tasks.ceph.osd.0:Started 2026-02-25T10:17:48.227 DEBUG:teuthology.orchestra.run.trial068:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-25T10:17:48.255 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T10:17:48.255 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T10:17:48.297 INFO:tasks.ceph.osd.1:Started 2026-02-25T10:17:48.297 DEBUG:teuthology.orchestra.run.trial072:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-25T10:17:48.323 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T10:17:48.323 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T10:17:48.365 INFO:tasks.ceph.osd.2:Started 2026-02-25T10:17:48.366 DEBUG:teuthology.orchestra.run.trial072:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2026-02-25T10:17:48.394 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-25T10:17:48.394 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2026-02-25T10:17:48.436 INFO:tasks.ceph.osd.3:Started 2026-02-25T10:17:48.436 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T10:17:48.513 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:48.509+0000 7f22f090b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:48.515 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:48.513+0000 7f22f090b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:48.549 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:48.549 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":7,"fsid":"e959f599-67ec-4a01-98b7-0bdf9a6b1fe1","created":"2026-02-25T10:17:46.267473+0000","modified":"2026-02-25T10:17:48.145683+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T10:17:48.145683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2ba36d3c-8fb4-4109-b054-75e9bf8b1a94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dc31ea2-f09f-4175-b190-bec35c045b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"699685d4-5d4f-4310-b2e5-4ce775e56238","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"459dac3b-8203-4a0d-8568-da567eec7c29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T10:17:48.558 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T10:17:48.559 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T10:17:48.962 DEBUG:teuthology.orchestra.run.trial068:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T10:17:49.028 INFO:teuthology.misc.health.trial068.stderr:2026-02-25T10:17:49.024+0000 7ff5d6dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:49.030 INFO:teuthology.misc.health.trial068.stderr:2026-02-25T10:17:49.028+0000 7ff5d6dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:49.063 INFO:teuthology.misc.health.trial068.stdout: 2026-02-25T10:17:49.063 INFO:teuthology.misc.health.trial068.stdout:{"epoch":7,"fsid":"e959f599-67ec-4a01-98b7-0bdf9a6b1fe1","created":"2026-02-25T10:17:46.267473+0000","modified":"2026-02-25T10:17:48.145683+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T10:17:48.145683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2ba36d3c-8fb4-4109-b054-75e9bf8b1a94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dc31ea2-f09f-4175-b190-bec35c045b2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"699685d4-5d4f-4310-b2e5-4ce775e56238","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"459dac3b-8203-4a0d-8568-da567eec7c29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T10:17:49.069 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-25T10:17:49.532 INFO:tasks.ceph.mgr.x.trial072.stderr:2026-02-25T10:17:49.530+0000 7f44b4856640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-25T10:17:55.472 DEBUG:teuthology.orchestra.run.trial068:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T10:17:55.542 INFO:teuthology.misc.health.trial068.stderr:2026-02-25T10:17:55.538+0000 7f877481a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:55.545 INFO:teuthology.misc.health.trial068.stderr:2026-02-25T10:17:55.543+0000 7f877481a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:55.657 INFO:teuthology.misc.health.trial068.stdout: 2026-02-25T10:17:55.657 INFO:teuthology.misc.health.trial068.stdout:{"epoch":11,"fsid":"e959f599-67ec-4a01-98b7-0bdf9a6b1fe1","created":"2026-02-25T10:17:46.267473+0000","modified":"2026-02-25T10:17:55.543494+0000","last_up_change":"2026-02-25T10:17:55.543494+0000","last_in_change":"2026-02-25T10:17:48.145683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2ba36d3c-8fb4-4109-b054-75e9bf8b1a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":2273297746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6805","nonce":2273297746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6809","nonce":2273297746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":2273297746}]},"public_addr":"10.20.193.68:6804/2273297746","cluster_addr":"10.20.193.68:6805/2273297746","heartbeat_back_addr":"10.20.193.68:6809/2273297746","heartbeat_front_addr":"10.20.193.68:6806/2273297746","state":["exists","up"]},{"osd":1,"uuid":"3dc31ea2-f09f-4175-b190-bec35c045b2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6807","nonce":3800132915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3800132915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3800132915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6803","nonce":3800132915}]},"public_addr":"10.20.193.68:6807/3800132915","cluster_addr":"10.20.193.68:6808/3800132915","heartbeat_back_addr":"10.20.193.68:6802/3800132915","heartbeat_front_addr":"10.20.193.68:6803/3800132915","state":["exists","up"]},{"osd":2,"uuid":"699685d4-5d4f-4310-b2e5-4ce775e56238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6807","nonce":895504062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6808","nonce":895504062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6804","nonce":895504062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6805","nonce":895504062}]},"public_addr":"10.20.193.72:6807/895504062","cluster_addr":"10.20.193.72:6808/895504062","heartbeat_back_addr":"10.20.193.72:6804/895504062","heartbeat_front_addr":"10.20.193.72:6805/895504062","state":["exists","up"]},{"osd":3,"uuid":"459dac3b-8203-4a0d-8568-da567eec7c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6810","nonce":2095571862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6811","nonce":2095571862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6803","nonce":2095571862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6802","nonce":2095571862}]},"public_addr":"10.20.193.72:6810/2095571862","cluster_addr":"10.20.193.72:6811/2095571862","heartbeat_back_addr":"10.20.193.72:6803/2095571862","heartbeat_front_addr":"10.20.193.72:6802/2095571862","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T10:17:55.664 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-25T10:17:55.664 INFO:tasks.ceph:Creating RBD pool 2026-02-25T10:17:55.664 DEBUG:teuthology.orchestra.run.trial068:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T10:17:55.739 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:55.733+0000 7f0e4ea11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:55.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:55.740+0000 7f0e4ea11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:56.549 INFO:teuthology.orchestra.run.trial068.stderr:pool 'rbd' created 2026-02-25T10:17:56.562 DEBUG:teuthology.orchestra.run.trial068:> rbd --cluster ceph pool init rbd 2026-02-25T10:17:56.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:56.601+0000 7fa7aba10240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:56.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:56.601+0000 7fa7aba10240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:56.606 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:56.605+0000 7fa7aba10240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:59.583 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T10:17:59.584 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T10:17:59.585 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T10:17:59.701 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:59.696+0000 7f6dc9da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:59.703 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:59.702+0000 7f6dc9da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:59.818 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:17:59.828 INFO:teuthology.orchestra.run.trial068.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-25T10:17:59.828 INFO:tasks.ceph_manager:config epoch is 1 2026-02-25T10:17:59.829 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T10:17:59.829 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T10:17:59.829 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T10:17:59.904 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:59.900+0000 7fa1530cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:17:59.907 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:17:59.905+0000 7fa1530cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.049 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:00.058 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6800","nonce":3550391897},{"type":"v1","addr":"10.20.193.72:6801","nonce":3550391897}]},"active_addr":"10.20.193.72:6801/3550391897","active_change":"2026-02-25T10:17:48.525422+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.72:0","nonce":977712507}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.72:0","nonce":879648335}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.72:0","nonce":3276488284}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.72:0","nonce":845621543}]}]} 2026-02-25T10:18:00.062 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T10:18:00.063 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T10:18:00.063 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T10:18:00.141 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.136+0000 7fb0092c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.143 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.141+0000 7fb0092c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.253 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:00.253 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":15,"fsid":"e959f599-67ec-4a01-98b7-0bdf9a6b1fe1","created":"2026-02-25T10:17:46.267473+0000","modified":"2026-02-25T10:17:59.554268+0000","last_up_change":"2026-02-25T10:17:55.543494+0000","last_in_change":"2026-02-25T10:17:48.145683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T10:17:55.853800+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-25T10:17:56.546216+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ba36d3c-8fb4-4109-b054-75e9bf8b1a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":2273297746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6805","nonce":2273297746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6809","nonce":2273297746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":2273297746}]},"public_addr":"10.20.193.68:6804/2273297746","cluster_addr":"10.20.193.68:6805/2273297746","heartbeat_back_addr":"10.20.193.68:6809/2273297746","heartbeat_front_addr":"10.20.193.68:6806/2273297746","state":["exists","up"]},{"osd":1,"uuid":"3dc31ea2-f09f-4175-b190-bec35c045b2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6807","nonce":3800132915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3800132915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3800132915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6803","nonce":3800132915}]},"public_addr":"10.20.193.68:6807/3800132915","cluster_addr":"10.20.193.68:6808/3800132915","heartbeat_back_addr":"10.20.193.68:6802/3800132915","heartbeat_front_addr":"10.20.193.68:6803/3800132915","state":["exists","up"]},{"osd":2,"uuid":"699685d4-5d4f-4310-b2e5-4ce775e56238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6807","nonce":895504062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6808","nonce":895504062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6804","nonce":895504062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6805","nonce":895504062}]},"public_addr":"10.20.193.72:6807/895504062","cluster_addr":"10.20.193.72:6808/895504062","heartbeat_back_addr":"10.20.193.72:6804/895504062","heartbeat_front_addr":"10.20.193.72:6805/895504062","state":["exists","up"]},{"osd":3,"uuid":"459dac3b-8203-4a0d-8568-da567eec7c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6810","nonce":2095571862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6811","nonce":2095571862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6803","nonce":2095571862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6802","nonce":2095571862}]},"public_addr":"10.20.193.72:6810/2095571862","cluster_addr":"10.20.193.72:6811/2095571862","heartbeat_back_addr":"10.20.193.72:6803/2095571862","heartbeat_front_addr":"10.20.193.72:6802/2095571862","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T10:18:00.262 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T10:18:00.262 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T10:18:00.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.332+0000 7f90a01a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.339 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.337+0000 7f90a01a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.449 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:00.449 INFO:teuthology.orchestra.run.trial068.stdout:{"epoch":15,"fsid":"e959f599-67ec-4a01-98b7-0bdf9a6b1fe1","created":"2026-02-25T10:17:46.267473+0000","modified":"2026-02-25T10:17:59.554268+0000","last_up_change":"2026-02-25T10:17:55.543494+0000","last_in_change":"2026-02-25T10:17:48.145683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T10:17:55.853800+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-25T10:17:56.546216+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ba36d3c-8fb4-4109-b054-75e9bf8b1a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":2273297746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6805","nonce":2273297746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6809","nonce":2273297746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":2273297746}]},"public_addr":"10.20.193.68:6804/2273297746","cluster_addr":"10.20.193.68:6805/2273297746","heartbeat_back_addr":"10.20.193.68:6809/2273297746","heartbeat_front_addr":"10.20.193.68:6806/2273297746","state":["exists","up"]},{"osd":1,"uuid":"3dc31ea2-f09f-4175-b190-bec35c045b2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6807","nonce":3800132915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":3800132915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":3800132915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6803","nonce":3800132915}]},"public_addr":"10.20.193.68:6807/3800132915","cluster_addr":"10.20.193.68:6808/3800132915","heartbeat_back_addr":"10.20.193.68:6802/3800132915","heartbeat_front_addr":"10.20.193.68:6803/3800132915","state":["exists","up"]},{"osd":2,"uuid":"699685d4-5d4f-4310-b2e5-4ce775e56238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6807","nonce":895504062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6808","nonce":895504062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6804","nonce":895504062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6805","nonce":895504062}]},"public_addr":"10.20.193.72:6807/895504062","cluster_addr":"10.20.193.72:6808/895504062","heartbeat_back_addr":"10.20.193.72:6804/895504062","heartbeat_front_addr":"10.20.193.72:6805/895504062","state":["exists","up"]},{"osd":3,"uuid":"459dac3b-8203-4a0d-8568-da567eec7c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6810","nonce":2095571862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6811","nonce":2095571862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6803","nonce":2095571862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.72:6802","nonce":2095571862}]},"public_addr":"10.20.193.72:6810/2095571862","cluster_addr":"10.20.193.72:6811/2095571862","heartbeat_back_addr":"10.20.193.72:6803/2095571862","heartbeat_front_addr":"10.20.193.72:6802/2095571862","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T10:18:00.458 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T10:18:00.458 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T10:18:00.459 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T10:18:00.459 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-25T10:18:00.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.535+0000 7f82afd83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.535+0000 7f2d4670b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.542 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.536+0000 7fa8aaecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.541+0000 7f82afd83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.541+0000 7f2d4670b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.539+0000 7f61c1583640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.544 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.543+0000 7fa8aaecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.546 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.545+0000 7f61c1583640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.565 INFO:teuthology.orchestra.run.trial068.stdout:42949672965 2026-02-25T10:18:00.566 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T10:18:00.567 INFO:teuthology.orchestra.run.trial068.stdout:47244640260 2026-02-25T10:18:00.567 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T10:18:00.568 INFO:teuthology.orchestra.run.trial068.stdout:42949672966 2026-02-25T10:18:00.569 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T10:18:00.570 INFO:teuthology.orchestra.run.trial068.stdout:47244640260 2026-02-25T10:18:00.570 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-25T10:18:00.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.683+0000 7f81f97d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.687 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.683+0000 7f0748d6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.688 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.683+0000 7fec186cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.689 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.685+0000 7f283e08b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.688+0000 7f0748d6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.690 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.689+0000 7f81f97d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.691 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.690+0000 7fec186cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.692 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.691+0000 7f283e08b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.805 INFO:teuthology.orchestra.run.trial068.stdout:42949672966 2026-02-25T10:18:00.806 INFO:teuthology.orchestra.run.trial068.stdout:47244640260 2026-02-25T10:18:00.810 INFO:teuthology.orchestra.run.trial068.stdout:42949672965 2026-02-25T10:18:00.815 INFO:teuthology.orchestra.run.trial068.stdout:47244640260 2026-02-25T10:18:00.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.2 2026-02-25T10:18:00.818 DEBUG:teuthology.parallel:result is None 2026-02-25T10:18:00.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2026-02-25T10:18:00.819 DEBUG:teuthology.parallel:result is None 2026-02-25T10:18:00.822 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672965 for osd.1 2026-02-25T10:18:00.823 DEBUG:teuthology.parallel:result is None 2026-02-25T10:18:00.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2026-02-25T10:18:00.826 DEBUG:teuthology.parallel:result is None 2026-02-25T10:18:00.827 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T10:18:00.827 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T10:18:00.942 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.938+0000 7f55c60cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:00.944 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:00.942+0000 7f55c60cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.061 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:01.061 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-25T10:18:01.070 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T10:18:00.557953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4128,"kb_used_data":4128,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662496,"statfs":{"total":213674622976,"available":213670395904,"internally_reserved":0,"allocated":4227072,"data_stored":4227072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567600+0000","last_change":"2026-02-25T10:17:59.570280+0000","last_active":"2026-02-25T10:17:59.567600+0000","last_peered":"2026-02-25T10:17:59.567600+0000","last_clean":"2026-02-25T10:17:59.567600+0000","last_became_active":"2026-02-25T10:17:57.603608+0000","last_became_peered":"2026-02-25T10:17:57.603608+0000","last_unstale":"2026-02-25T10:17:59.567600+0000","last_undegraded":"2026-02-25T10:17:59.567600+0000","last_fullsized":"2026-02-25T10:17:59.567600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567661+0000","last_change":"2026-02-25T10:17:59.569854+0000","last_active":"2026-02-25T10:17:59.567661+0000","last_peered":"2026-02-25T10:17:59.567661+0000","last_clean":"2026-02-25T10:17:59.567661+0000","last_became_active":"2026-02-25T10:17:57.594227+0000","last_became_peered":"2026-02-25T10:17:57.594227+0000","last_unstale":"2026-02-25T10:17:59.567661+0000","last_undegraded":"2026-02-25T10:17:59.567661+0000","last_fullsized":"2026-02-25T10:17:59.567661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567832+0000","last_change":"2026-02-25T10:17:59.570677+0000","last_active":"2026-02-25T10:17:59.567832+0000","last_peered":"2026-02-25T10:17:59.567832+0000","last_clean":"2026-02-25T10:17:59.567832+0000","last_became_active":"2026-02-25T10:17:57.592795+0000","last_became_peered":"2026-02-25T10:17:57.592795+0000","last_unstale":"2026-02-25T10:17:59.567832+0000","last_undegraded":"2026-02-25T10:17:59.567832+0000","last_fullsized":"2026-02-25T10:17:59.567832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.566066+0000","last_change":"2026-02-25T10:17:59.568986+0000","last_active":"2026-02-25T10:17:59.566066+0000","last_peered":"2026-02-25T10:17:59.566066+0000","last_clean":"2026-02-25T10:17:59.566066+0000","last_became_active":"2026-02-25T10:17:57.594095+0000","last_became_peered":"2026-02-25T10:17:57.594095+0000","last_unstale":"2026-02-25T10:17:59.566066+0000","last_undegraded":"2026-02-25T10:17:59.566066+0000","last_fullsized":"2026-02-25T10:17:59.566066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568519+0000","last_change":"2026-02-25T10:17:57.612424+0000","last_active":"2026-02-25T10:17:59.568519+0000","last_peered":"2026-02-25T10:17:59.568519+0000","last_clean":"2026-02-25T10:17:59.568519+0000","last_became_active":"2026-02-25T10:17:57.597495+0000","last_became_peered":"2026-02-25T10:17:57.597495+0000","last_unstale":"2026-02-25T10:17:59.568519+0000","last_undegraded":"2026-02-25T10:17:59.568519+0000","last_fullsized":"2026-02-25T10:17:59.568519+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567310+0000","last_change":"2026-02-25T10:17:59.569701+0000","last_active":"2026-02-25T10:17:59.567310+0000","last_peered":"2026-02-25T10:17:59.567310+0000","last_clean":"2026-02-25T10:17:59.567310+0000","last_became_active":"2026-02-25T10:17:57.594174+0000","last_became_peered":"2026-02-25T10:17:57.594174+0000","last_unstale":"2026-02-25T10:17:59.567310+0000","last_undegraded":"2026-02-25T10:17:59.567310+0000","last_fullsized":"2026-02-25T10:17:59.567310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568503+0000","last_change":"2026-02-25T10:17:59.568503+0000","last_active":"2026-02-25T10:17:59.568503+0000","last_peered":"2026-02-25T10:17:59.568503+0000","last_clean":"2026-02-25T10:17:59.568503+0000","last_became_active":"2026-02-25T10:17:57.607343+0000","last_became_peered":"2026-02-25T10:17:57.607343+0000","last_unstale":"2026-02-25T10:17:59.568503+0000","last_undegraded":"2026-02-25T10:17:59.568503+0000","last_fullsized":"2026-02-25T10:17:59.568503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.565888+0000","last_change":"2026-02-25T10:17:59.569065+0000","last_active":"2026-02-25T10:17:59.565888+0000","last_peered":"2026-02-25T10:17:59.565888+0000","last_clean":"2026-02-25T10:17:59.565888+0000","last_became_active":"2026-02-25T10:17:57.593096+0000","last_became_peered":"2026-02-25T10:17:57.593096+0000","last_unstale":"2026-02-25T10:17:59.565888+0000","last_undegraded":"2026-02-25T10:17:59.565888+0000","last_fullsized":"2026-02-25T10:17:59.565888+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.564834+0000","last_change":"2026-02-25T10:17:59.568550+0000","last_active":"2026-02-25T10:17:59.564834+0000","last_peered":"2026-02-25T10:17:59.564834+0000","last_clean":"2026-02-25T10:17:59.564834+0000","last_became_active":"2026-02-25T10:17:57.614238+0000","last_became_peered":"2026-02-25T10:17:57.614238+0000","last_unstale":"2026-02-25T10:17:59.564834+0000","last_undegraded":"2026-02-25T10:17:59.564834+0000","last_fullsized":"2026-02-25T10:17:59.564834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670371328,"internally_reserved":0,"allocated":4251648,"data_stored":4251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1236,"kb_used_data":1236,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165420,"statfs":{"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1228,"kb_used_data":1228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165428,"statfs":{"total":53418655744,"available":53417398272,"internally_reserved":0,"allocated":1257472,"data_stored":1257472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1224,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165432,"statfs":{"total":53418655744,"available":53417402368,"internally_reserved":0,"allocated":1253376,"data_stored":1253376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53418655744,"available":53418196992,"internally_reserved":0,"allocated":458752,"data_stored":458752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T10:18:01.071 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T10:18:01.145 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.141+0000 7fc11c9fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.147 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.145+0000 7fc11c9fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.259 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:01.259 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-25T10:18:01.268 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T10:18:00.557953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4128,"kb_used_data":4128,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662496,"statfs":{"total":213674622976,"available":213670395904,"internally_reserved":0,"allocated":4227072,"data_stored":4227072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567600+0000","last_change":"2026-02-25T10:17:59.570280+0000","last_active":"2026-02-25T10:17:59.567600+0000","last_peered":"2026-02-25T10:17:59.567600+0000","last_clean":"2026-02-25T10:17:59.567600+0000","last_became_active":"2026-02-25T10:17:57.603608+0000","last_became_peered":"2026-02-25T10:17:57.603608+0000","last_unstale":"2026-02-25T10:17:59.567600+0000","last_undegraded":"2026-02-25T10:17:59.567600+0000","last_fullsized":"2026-02-25T10:17:59.567600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567661+0000","last_change":"2026-02-25T10:17:59.569854+0000","last_active":"2026-02-25T10:17:59.567661+0000","last_peered":"2026-02-25T10:17:59.567661+0000","last_clean":"2026-02-25T10:17:59.567661+0000","last_became_active":"2026-02-25T10:17:57.594227+0000","last_became_peered":"2026-02-25T10:17:57.594227+0000","last_unstale":"2026-02-25T10:17:59.567661+0000","last_undegraded":"2026-02-25T10:17:59.567661+0000","last_fullsized":"2026-02-25T10:17:59.567661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567832+0000","last_change":"2026-02-25T10:17:59.570677+0000","last_active":"2026-02-25T10:17:59.567832+0000","last_peered":"2026-02-25T10:17:59.567832+0000","last_clean":"2026-02-25T10:17:59.567832+0000","last_became_active":"2026-02-25T10:17:57.592795+0000","last_became_peered":"2026-02-25T10:17:57.592795+0000","last_unstale":"2026-02-25T10:17:59.567832+0000","last_undegraded":"2026-02-25T10:17:59.567832+0000","last_fullsized":"2026-02-25T10:17:59.567832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.566066+0000","last_change":"2026-02-25T10:17:59.568986+0000","last_active":"2026-02-25T10:17:59.566066+0000","last_peered":"2026-02-25T10:17:59.566066+0000","last_clean":"2026-02-25T10:17:59.566066+0000","last_became_active":"2026-02-25T10:17:57.594095+0000","last_became_peered":"2026-02-25T10:17:57.594095+0000","last_unstale":"2026-02-25T10:17:59.566066+0000","last_undegraded":"2026-02-25T10:17:59.566066+0000","last_fullsized":"2026-02-25T10:17:59.566066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568519+0000","last_change":"2026-02-25T10:17:57.612424+0000","last_active":"2026-02-25T10:17:59.568519+0000","last_peered":"2026-02-25T10:17:59.568519+0000","last_clean":"2026-02-25T10:17:59.568519+0000","last_became_active":"2026-02-25T10:17:57.597495+0000","last_became_peered":"2026-02-25T10:17:57.597495+0000","last_unstale":"2026-02-25T10:17:59.568519+0000","last_undegraded":"2026-02-25T10:17:59.568519+0000","last_fullsized":"2026-02-25T10:17:59.568519+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567310+0000","last_change":"2026-02-25T10:17:59.569701+0000","last_active":"2026-02-25T10:17:59.567310+0000","last_peered":"2026-02-25T10:17:59.567310+0000","last_clean":"2026-02-25T10:17:59.567310+0000","last_became_active":"2026-02-25T10:17:57.594174+0000","last_became_peered":"2026-02-25T10:17:57.594174+0000","last_unstale":"2026-02-25T10:17:59.567310+0000","last_undegraded":"2026-02-25T10:17:59.567310+0000","last_fullsized":"2026-02-25T10:17:59.567310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568503+0000","last_change":"2026-02-25T10:17:59.568503+0000","last_active":"2026-02-25T10:17:59.568503+0000","last_peered":"2026-02-25T10:17:59.568503+0000","last_clean":"2026-02-25T10:17:59.568503+0000","last_became_active":"2026-02-25T10:17:57.607343+0000","last_became_peered":"2026-02-25T10:17:57.607343+0000","last_unstale":"2026-02-25T10:17:59.568503+0000","last_undegraded":"2026-02-25T10:17:59.568503+0000","last_fullsized":"2026-02-25T10:17:59.568503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.565888+0000","last_change":"2026-02-25T10:17:59.569065+0000","last_active":"2026-02-25T10:17:59.565888+0000","last_peered":"2026-02-25T10:17:59.565888+0000","last_clean":"2026-02-25T10:17:59.565888+0000","last_became_active":"2026-02-25T10:17:57.593096+0000","last_became_peered":"2026-02-25T10:17:57.593096+0000","last_unstale":"2026-02-25T10:17:59.565888+0000","last_undegraded":"2026-02-25T10:17:59.565888+0000","last_fullsized":"2026-02-25T10:17:59.565888+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.564834+0000","last_change":"2026-02-25T10:17:59.568550+0000","last_active":"2026-02-25T10:17:59.564834+0000","last_peered":"2026-02-25T10:17:59.564834+0000","last_clean":"2026-02-25T10:17:59.564834+0000","last_became_active":"2026-02-25T10:17:57.614238+0000","last_became_peered":"2026-02-25T10:17:57.614238+0000","last_unstale":"2026-02-25T10:17:59.564834+0000","last_undegraded":"2026-02-25T10:17:59.564834+0000","last_fullsized":"2026-02-25T10:17:59.564834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670371328,"internally_reserved":0,"allocated":4251648,"data_stored":4251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1236,"kb_used_data":1236,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165420,"statfs":{"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1228,"kb_used_data":1228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165428,"statfs":{"total":53418655744,"available":53417398272,"internally_reserved":0,"allocated":1257472,"data_stored":1257472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1224,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165432,"statfs":{"total":53418655744,"available":53417402368,"internally_reserved":0,"allocated":1253376,"data_stored":1253376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53418655744,"available":53418196992,"internally_reserved":0,"allocated":458752,"data_stored":458752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T10:18:01.270 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T10:18:01.270 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T10:18:01.270 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T10:18:01.270 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T10:18:01.346 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.342+0000 7f28d3d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.348 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.346+0000 7f28d3d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.485 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:01.485 INFO:teuthology.orchestra.run.trial068.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T10:18:01.494 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T10:18:01.495 INFO:teuthology.run_tasks:Running task thrashosds... 2026-02-25T10:18:01.514 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T10:18:01.514 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T10:18:01.514 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T10:18:01.514 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2026-02-25T10:18:01.592 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.586+0000 7f8f05011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.594 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.593+0000 7f8f05011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.718 INFO:tasks.thrashosds:Beginning thrashosds... 2026-02-25T10:18:01.718 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T10:18:01.718 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T10:18:01.793 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.789+0000 7f797945a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.794+0000 7f797945a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:01.910 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:01.910 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-25T10:18:01.919 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T10:18:00.557953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4128,"kb_used_data":4128,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662496,"statfs":{"total":213674622976,"available":213670395904,"internally_reserved":0,"allocated":4227072,"data_stored":4227072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567600+0000","last_change":"2026-02-25T10:17:59.570280+0000","last_active":"2026-02-25T10:17:59.567600+0000","last_peered":"2026-02-25T10:17:59.567600+0000","last_clean":"2026-02-25T10:17:59.567600+0000","last_became_active":"2026-02-25T10:17:57.603608+0000","last_became_peered":"2026-02-25T10:17:57.603608+0000","last_unstale":"2026-02-25T10:17:59.567600+0000","last_undegraded":"2026-02-25T10:17:59.567600+0000","last_fullsized":"2026-02-25T10:17:59.567600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567661+0000","last_change":"2026-02-25T10:17:59.569854+0000","last_active":"2026-02-25T10:17:59.567661+0000","last_peered":"2026-02-25T10:17:59.567661+0000","last_clean":"2026-02-25T10:17:59.567661+0000","last_became_active":"2026-02-25T10:17:57.594227+0000","last_became_peered":"2026-02-25T10:17:57.594227+0000","last_unstale":"2026-02-25T10:17:59.567661+0000","last_undegraded":"2026-02-25T10:17:59.567661+0000","last_fullsized":"2026-02-25T10:17:59.567661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567832+0000","last_change":"2026-02-25T10:17:59.570677+0000","last_active":"2026-02-25T10:17:59.567832+0000","last_peered":"2026-02-25T10:17:59.567832+0000","last_clean":"2026-02-25T10:17:59.567832+0000","last_became_active":"2026-02-25T10:17:57.592795+0000","last_became_peered":"2026-02-25T10:17:57.592795+0000","last_unstale":"2026-02-25T10:17:59.567832+0000","last_undegraded":"2026-02-25T10:17:59.567832+0000","last_fullsized":"2026-02-25T10:17:59.567832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.566066+0000","last_change":"2026-02-25T10:17:59.568986+0000","last_active":"2026-02-25T10:17:59.566066+0000","last_peered":"2026-02-25T10:17:59.566066+0000","last_clean":"2026-02-25T10:17:59.566066+0000","last_became_active":"2026-02-25T10:17:57.594095+0000","last_became_peered":"2026-02-25T10:17:57.594095+0000","last_unstale":"2026-02-25T10:17:59.566066+0000","last_undegraded":"2026-02-25T10:17:59.566066+0000","last_fullsized":"2026-02-25T10:17:59.566066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568519+0000","last_change":"2026-02-25T10:17:57.612424+0000","last_active":"2026-02-25T10:17:59.568519+0000","last_peered":"2026-02-25T10:17:59.568519+0000","last_clean":"2026-02-25T10:17:59.568519+0000","last_became_active":"2026-02-25T10:17:57.597495+0000","last_became_peered":"2026-02-25T10:17:57.597495+0000","last_unstale":"2026-02-25T10:17:59.568519+0000","last_undegraded":"2026-02-25T10:17:59.568519+0000","last_fullsized":"2026-02-25T10:17:59.568519+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567310+0000","last_change":"2026-02-25T10:17:59.569701+0000","last_active":"2026-02-25T10:17:59.567310+0000","last_peered":"2026-02-25T10:17:59.567310+0000","last_clean":"2026-02-25T10:17:59.567310+0000","last_became_active":"2026-02-25T10:17:57.594174+0000","last_became_peered":"2026-02-25T10:17:57.594174+0000","last_unstale":"2026-02-25T10:17:59.567310+0000","last_undegraded":"2026-02-25T10:17:59.567310+0000","last_fullsized":"2026-02-25T10:17:59.567310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568503+0000","last_change":"2026-02-25T10:17:59.568503+0000","last_active":"2026-02-25T10:17:59.568503+0000","last_peered":"2026-02-25T10:17:59.568503+0000","last_clean":"2026-02-25T10:17:59.568503+0000","last_became_active":"2026-02-25T10:17:57.607343+0000","last_became_peered":"2026-02-25T10:17:57.607343+0000","last_unstale":"2026-02-25T10:17:59.568503+0000","last_undegraded":"2026-02-25T10:17:59.568503+0000","last_fullsized":"2026-02-25T10:17:59.568503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.565888+0000","last_change":"2026-02-25T10:17:59.569065+0000","last_active":"2026-02-25T10:17:59.565888+0000","last_peered":"2026-02-25T10:17:59.565888+0000","last_clean":"2026-02-25T10:17:59.565888+0000","last_became_active":"2026-02-25T10:17:57.593096+0000","last_became_peered":"2026-02-25T10:17:57.593096+0000","last_unstale":"2026-02-25T10:17:59.565888+0000","last_undegraded":"2026-02-25T10:17:59.565888+0000","last_fullsized":"2026-02-25T10:17:59.565888+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.564834+0000","last_change":"2026-02-25T10:17:59.568550+0000","last_active":"2026-02-25T10:17:59.564834+0000","last_peered":"2026-02-25T10:17:59.564834+0000","last_clean":"2026-02-25T10:17:59.564834+0000","last_became_active":"2026-02-25T10:17:57.614238+0000","last_became_peered":"2026-02-25T10:17:57.614238+0000","last_unstale":"2026-02-25T10:17:59.564834+0000","last_undegraded":"2026-02-25T10:17:59.564834+0000","last_fullsized":"2026-02-25T10:17:59.564834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670371328,"internally_reserved":0,"allocated":4251648,"data_stored":4251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1236,"kb_used_data":1236,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165420,"statfs":{"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1228,"kb_used_data":1228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165428,"statfs":{"total":53418655744,"available":53417398272,"internally_reserved":0,"allocated":1257472,"data_stored":1257472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1224,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165432,"statfs":{"total":53418655744,"available":53417402368,"internally_reserved":0,"allocated":1253376,"data_stored":1253376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53418655744,"available":53418196992,"internally_reserved":0,"allocated":458752,"data_stored":458752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T10:18:01.921 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T10:18:01.998 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.993+0000 7f84ec411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.000 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:01.998+0000 7f84ec411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.110 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:02.110 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-25T10:18:02.120 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T10:18:00.557953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4128,"kb_used_data":4128,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662496,"statfs":{"total":213674622976,"available":213670395904,"internally_reserved":0,"allocated":4227072,"data_stored":4227072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567600+0000","last_change":"2026-02-25T10:17:59.570280+0000","last_active":"2026-02-25T10:17:59.567600+0000","last_peered":"2026-02-25T10:17:59.567600+0000","last_clean":"2026-02-25T10:17:59.567600+0000","last_became_active":"2026-02-25T10:17:57.603608+0000","last_became_peered":"2026-02-25T10:17:57.603608+0000","last_unstale":"2026-02-25T10:17:59.567600+0000","last_undegraded":"2026-02-25T10:17:59.567600+0000","last_fullsized":"2026-02-25T10:17:59.567600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567661+0000","last_change":"2026-02-25T10:17:59.569854+0000","last_active":"2026-02-25T10:17:59.567661+0000","last_peered":"2026-02-25T10:17:59.567661+0000","last_clean":"2026-02-25T10:17:59.567661+0000","last_became_active":"2026-02-25T10:17:57.594227+0000","last_became_peered":"2026-02-25T10:17:57.594227+0000","last_unstale":"2026-02-25T10:17:59.567661+0000","last_undegraded":"2026-02-25T10:17:59.567661+0000","last_fullsized":"2026-02-25T10:17:59.567661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567832+0000","last_change":"2026-02-25T10:17:59.570677+0000","last_active":"2026-02-25T10:17:59.567832+0000","last_peered":"2026-02-25T10:17:59.567832+0000","last_clean":"2026-02-25T10:17:59.567832+0000","last_became_active":"2026-02-25T10:17:57.592795+0000","last_became_peered":"2026-02-25T10:17:57.592795+0000","last_unstale":"2026-02-25T10:17:59.567832+0000","last_undegraded":"2026-02-25T10:17:59.567832+0000","last_fullsized":"2026-02-25T10:17:59.567832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.566066+0000","last_change":"2026-02-25T10:17:59.568986+0000","last_active":"2026-02-25T10:17:59.566066+0000","last_peered":"2026-02-25T10:17:59.566066+0000","last_clean":"2026-02-25T10:17:59.566066+0000","last_became_active":"2026-02-25T10:17:57.594095+0000","last_became_peered":"2026-02-25T10:17:57.594095+0000","last_unstale":"2026-02-25T10:17:59.566066+0000","last_undegraded":"2026-02-25T10:17:59.566066+0000","last_fullsized":"2026-02-25T10:17:59.566066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568519+0000","last_change":"2026-02-25T10:17:57.612424+0000","last_active":"2026-02-25T10:17:59.568519+0000","last_peered":"2026-02-25T10:17:59.568519+0000","last_clean":"2026-02-25T10:17:59.568519+0000","last_became_active":"2026-02-25T10:17:57.597495+0000","last_became_peered":"2026-02-25T10:17:57.597495+0000","last_unstale":"2026-02-25T10:17:59.568519+0000","last_undegraded":"2026-02-25T10:17:59.568519+0000","last_fullsized":"2026-02-25T10:17:59.568519+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567310+0000","last_change":"2026-02-25T10:17:59.569701+0000","last_active":"2026-02-25T10:17:59.567310+0000","last_peered":"2026-02-25T10:17:59.567310+0000","last_clean":"2026-02-25T10:17:59.567310+0000","last_became_active":"2026-02-25T10:17:57.594174+0000","last_became_peered":"2026-02-25T10:17:57.594174+0000","last_unstale":"2026-02-25T10:17:59.567310+0000","last_undegraded":"2026-02-25T10:17:59.567310+0000","last_fullsized":"2026-02-25T10:17:59.567310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568503+0000","last_change":"2026-02-25T10:17:59.568503+0000","last_active":"2026-02-25T10:17:59.568503+0000","last_peered":"2026-02-25T10:17:59.568503+0000","last_clean":"2026-02-25T10:17:59.568503+0000","last_became_active":"2026-02-25T10:17:57.607343+0000","last_became_peered":"2026-02-25T10:17:57.607343+0000","last_unstale":"2026-02-25T10:17:59.568503+0000","last_undegraded":"2026-02-25T10:17:59.568503+0000","last_fullsized":"2026-02-25T10:17:59.568503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.565888+0000","last_change":"2026-02-25T10:17:59.569065+0000","last_active":"2026-02-25T10:17:59.565888+0000","last_peered":"2026-02-25T10:17:59.565888+0000","last_clean":"2026-02-25T10:17:59.565888+0000","last_became_active":"2026-02-25T10:17:57.593096+0000","last_became_peered":"2026-02-25T10:17:57.593096+0000","last_unstale":"2026-02-25T10:17:59.565888+0000","last_undegraded":"2026-02-25T10:17:59.565888+0000","last_fullsized":"2026-02-25T10:17:59.565888+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.564834+0000","last_change":"2026-02-25T10:17:59.568550+0000","last_active":"2026-02-25T10:17:59.564834+0000","last_peered":"2026-02-25T10:17:59.564834+0000","last_clean":"2026-02-25T10:17:59.564834+0000","last_became_active":"2026-02-25T10:17:57.614238+0000","last_became_peered":"2026-02-25T10:17:57.614238+0000","last_unstale":"2026-02-25T10:17:59.564834+0000","last_undegraded":"2026-02-25T10:17:59.564834+0000","last_fullsized":"2026-02-25T10:17:59.564834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670371328,"internally_reserved":0,"allocated":4251648,"data_stored":4251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1236,"kb_used_data":1236,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165420,"statfs":{"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1228,"kb_used_data":1228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165428,"statfs":{"total":53418655744,"available":53417398272,"internally_reserved":0,"allocated":1257472,"data_stored":1257472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1224,"kb_used_data":1224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165432,"statfs":{"total":53418655744,"available":53417402368,"internally_reserved":0,"allocated":1253376,"data_stored":1253376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53418655744,"available":53418196992,"internally_reserved":0,"allocated":458752,"data_stored":458752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T10:18:02.120 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T10:18:02.120 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-25T10:18:02.194 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.190+0000 7f4f926cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.197 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.195+0000 7f4f926cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.308 INFO:teuthology.orchestra.run.trial068.stdout:epoch 15 2026-02-25T10:18:02.308 INFO:teuthology.orchestra.run.trial068.stdout:fsid e959f599-67ec-4a01-98b7-0bdf9a6b1fe1 2026-02-25T10:18:02.308 INFO:teuthology.orchestra.run.trial068.stdout:created 2026-02-25T10:17:46.267473+0000 2026-02-25T10:18:02.308 INFO:teuthology.orchestra.run.trial068.stdout:modified 2026-02-25T10:17:59.554268+0000 2026-02-25T10:18:02.308 INFO:teuthology.orchestra.run.trial068.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-25T10:18:02.308 INFO:teuthology.orchestra.run.trial068.stdout:crush_version 5 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:full_ratio 0.95 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:backfillfull_ratio 0.9 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:nearfull_ratio 0.85 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:require_min_compat_client luminous 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:min_compat_client jewel 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:require_osd_release tentacle 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:stretch_mode_enabled false 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:allow_crimson=true 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 15 flags hashpspool,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 1.50 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout: removed_snaps_queue [2~1] 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:pool 2 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:max_osd 4 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:osd.0 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.68:6804/2273297746 v2:10.20.193.68:6805/2273297746 exists,up 2ba36d3c-8fb4-4109-b054-75e9bf8b1a94 2026-02-25T10:18:02.309 INFO:teuthology.orchestra.run.trial068.stdout:osd.1 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.68:6807/3800132915 v2:10.20.193.68:6808/3800132915 exists,up 3dc31ea2-f09f-4175-b190-bec35c045b2c 2026-02-25T10:18:02.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.2 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.72:6807/895504062 v2:10.20.193.72:6808/895504062 exists,up 699685d4-5d4f-4310-b2e5-4ce775e56238 2026-02-25T10:18:02.310 INFO:teuthology.orchestra.run.trial068.stdout:osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.72:6810/2095571862 v2:10.20.193.72:6811/2095571862 exists,up 459dac3b-8203-4a0d-8568-da567eec7c29 2026-02-25T10:18:02.317 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.68:6804/2273297746 v2:10.20.193.68:6805/2273297746 exists,up 2ba36d3c-8fb4-4109-b054-75e9bf8b1a94', 'osd.1 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.68:6807/3800132915 v2:10.20.193.68:6808/3800132915 exists,up 3dc31ea2-f09f-4175-b190-bec35c045b2c', 'osd.2 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.72:6807/895504062 v2:10.20.193.72:6808/895504062 exists,up 699685d4-5d4f-4310-b2e5-4ce775e56238', 'osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.72:6810/2095571862 v2:10.20.193.72:6811/2095571862 exists,up 459dac3b-8203-4a0d-8568-da567eec7c29'] 2026-02-25T10:18:02.318 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2026-02-25T10:18:02.392 INFO:teuthology.orchestra.run.trial068.stdout:{ 2026-02-25T10:18:02.392 INFO:teuthology.orchestra.run.trial068.stdout: "name": "mon.a", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "cluster": "ceph", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "admin_socket_mode": "", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "auth_client_required": "cephx, none", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "auth_cluster_required": "cephx", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "auth_debug": "true", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "auth_mon_ticket_ttl": "660.000000", 2026-02-25T10:18:02.393 INFO:teuthology.orchestra.run.trial068.stdout: "auth_service_required": "cephx", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "auth_service_ticket_ttl": "240.000000", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "auth_supported": "cephx", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio": "true", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_max_queue_depth": "1024", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_poll_ms": "250", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_reap_max": "16", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_submit_retry_initial_delay_us": "125", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_aio_submit_retry_max": "16", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_async_discard": "false", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_async_discard_max_pending": "1000000", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_async_discard_threads": "0", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_block_size": "4096", 2026-02-25T10:18:02.394 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_aio": "false", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_aio_log_age": "5.000000", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_discard_sleep": "0", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_debug_inflight_ios": "false", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_discard_max_bytes": "10737418240", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_enable_discard": "false", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_flock_retry": "3", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_flock_retry_interval": "0.100000", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_inject_crash": "0", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_inject_crash_flush_delay": "2", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_ioring": "false", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_ioring_hipri": "false", 2026-02-25T10:18:02.395 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_ioring_sqthread_poll": "false", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_max_discard_length": "2147483648", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_nvme_unbind_from_kernel": "false", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_read_buffer_alignment": "4096", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_read_preallocated_huge_buffers": "", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_stalled_read_warn_threshold": "1", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bdev_type": "", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_alloc_size": "1048576", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_allocator": "hybrid", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_buffered_io": "true", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_check_for_zeros": "false", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_check_volume_selector_often": "false", 2026-02-25T10:18:02.396 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_check_volume_selector_on_umount": "false", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_compact_log_sync": "false", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_log_compact_min_size": "16777216", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_log_replay_check_allocations": "true", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_max_log_runway": "4194304", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_max_prefetch": "1048576", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_min_flush_size": "524288", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_min_log_runway": "1048576", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_replay_recovery": "false", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_replay_recovery_disable_compact": "false", 2026-02-25T10:18:02.397 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_shared_alloc_size": "65536", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_sync_write": "false", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluefs_wal_envelope_mode": "true", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_allocation_from_file": "true", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_allocator": "hybrid", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_allocator_lookup_policy": "auto", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_async_db_compaction": "true", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2026-02-25T10:18:02.398 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bdev_label_multi": "true", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bdev_label_require_all": "true", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bitmapallocator_span_size": "1024", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_blobid_prealloc": "10240", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_create": "true", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_db_create": "false", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_db_path": "", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_db_size": "0", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_path": "", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_preallocate_file": "false", 2026-02-25T10:18:02.399 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_size": "107374182400", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_wal_create": "false", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_wal_path": "", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_block_wal_size": "100663296", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs": "true", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs_env_mirror": "false", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_bluefs_max_free": "10737418240", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2026-02-25T10:18:02.400 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_autotune": "true", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_autotune_interval": "5.000000", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_kv_ratio": "0.450000", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_meta_ratio": "0.450000", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_size": "0", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_size_hdd": "1073741824", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_size_ssd": "3221225472", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_trim_interval": "0.050000", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cache_type": "2q", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2026-02-25T10:18:02.401 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_clone_cow": "true", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_algorithm": "snappy", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_max_blob_size": "0", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_min_blob_size": "0", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_mode": "none", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_compression_required_ratio": "0.875000", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_csum_type": "crc32c", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_enforce_settings": "default", 2026-02-25T10:18:02.402 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_extent_map_encode_check": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_freelist": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_fsck_abort": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_inject_read_err": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_legacy_omap": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_no_reuse_blocks": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_omit_block_device_write": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_omit_kv_commit": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_onode_segmentation_random": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_prefragment_max": "1048576", 2026-02-25T10:18:02.403 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_random_read_err": "0.000000", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_small_allocations": "0", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_default_buffered_read": "true", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_default_buffered_write": "false", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_deferred_batch_ops": "0", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_discard_on_mkfs": "true", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_elastic_shared_blobs": "true", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_max_size": "1200", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_min_size": "150", 2026-02-25T10:18:02.404 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_target_size": "500", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fail_eio": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fragmentation_check_period": "3600", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_freelist_blocks_per_key": "128", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mkfs": "true", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mount": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_mount_deep": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_umount": "false", 2026-02-25T10:18:02.405 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_on_umount_deep": "false", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_quick_fix_threads": "2", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_gc_enable_blob_threshold": "0", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_gc_enable_total_threshold": "0", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_ignore_data_csum": "false", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_kvbackend": "rocksdb", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_log_collection_list_age": "60.000000", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2026-02-25T10:18:02.406 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_log_op_age": "5.000000", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_alloc_size": "0", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_blob_size": "0", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_blob_size_hdd": "65536", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_blob_size_ssd": "65536", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_defer_interval": "3.000000", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_max_deferred_txc": "32", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_min_alloc_size": "0", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_min_alloc_size_hdd": "4096", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_min_alloc_size_ssd": "4096", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_nid_prealloc": "1024", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_onode_segment_size": "0", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_prefer_deferred_size": "0", 2026-02-25T10:18:02.407 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_qfsck_on_mount": "true", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_recompression_min_gain": "1.200000", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_retry_disk_reads": "3", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_cf": "true", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_rocksdb_options_annex": "", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_slow_ops_warn_threshold": "1", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_coremask": "0x1", 2026-02-25T10:18:02.408 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_io_sleep": "5", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_max_io_completion": "0", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_spdk_mem": "512", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_sync_submit_transaction": "false", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_bytes": "67108864", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_cost_per_io": "0", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_throttle_trace_rate": "0.000000", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_tracing": "false", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2026-02-25T10:18:02.409 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_volume_selection_reserved": "0", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_free_fragmentation": "0.800000", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_legacy_statfs": "true", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_write_v2": "false", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_write_v2_random": "false", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "bluestore_zero_block_detection": "false", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "breakpad": "false", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "ceph_assert_supresssions": "", 2026-02-25T10:18:02.410 INFO:teuthology.orchestra.run.trial068.stdout: "cephadm_path": "/usr/sbin/cephadm", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_action_update_interval": "2", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_directory_scan_interval": "10", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_mount_timeout": "10", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_perf_stats_prio": "5", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephsqlite_blocklist_dead_locker": "true", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephsqlite_lock_renewal_interval": "2000", 2026-02-25T10:18:02.411 INFO:teuthology.orchestra.run.trial068.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_cluster_require_signatures": "false", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_cluster_require_version": "2", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_require_signatures": "false", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_require_version": "2", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_service_require_signatures": "false", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_service_require_version": "2", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "cephx_sign_messages": "true", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "chdir": "", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "client_acl_type": "", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "client_asio_thread_count": "2", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "client_cache_mid": "0.750000", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "client_cache_size": "16384", 2026-02-25T10:18:02.412 INFO:teuthology.orchestra.run.trial068.stdout: "client_caps_release_delay": "5", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_check_pool_perm": "true", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_collect_and_send_global_metrics": "false", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_force_sync_read": "false", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_getattr_caps": "false", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_inject_features": "", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_debug_inject_tick_delay": "0", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_die_on_failed_dentry_invalidate": "true", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_die_on_failed_remount": "false", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_dirsize_rbytes": "true", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_force_lazyio": "false", 2026-02-25T10:18:02.413 INFO:teuthology.orchestra.run.trial068.stdout: "client_fs": "", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_fscrypt_as": "true", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_fscrypt_dummy_encryption": "false", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_inject_fixed_oldest_tid": "false", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_inject_release_failure": "false", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_inject_write_delay_secs": "0", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_max_inline_size": "4096", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_max_retries_on_remount_failure": "5", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_mds_namespace": "", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_metadata": "", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_mount_gid": "-1", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_mount_timeout": "300", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_mount_uid": "-1", 2026-02-25T10:18:02.414 INFO:teuthology.orchestra.run.trial068.stdout: "client_mountpoint": "/", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_notify_timeout": "10", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc": "true", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_max_dirty": "104857600", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_max_dirty_age": "5.000000", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_max_objects": "1000", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_size": "209715200", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_oc_target_dirty": "8388608", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_permissions": "true", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_quota": "true", 2026-02-25T10:18:02.415 INFO:teuthology.orchestra.run.trial068.stdout: "client_quota_df": "true", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_readahead_max_bytes": "0", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_readahead_max_periods": "4", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_readahead_min": "131072", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_reconnect_stale": "false", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_respect_subvolume_snapshot_visibility": "false", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_shutdown_timeout": "30", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_snapdir": ".snap", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_tick_interval": "1", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_trace": "", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_try_dentry_invalidate": "false", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_use_faked_inos": "false", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "client_use_random_mds": "false", 2026-02-25T10:18:02.416 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_graylog": "false", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_graylog_host": "127.0.0.1", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_graylog_port": "12201", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_monitors": "default=true", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_syslog": "false", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "clog_to_syslog_level": "info", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "cluster_addr": "-", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "cluster_network": "", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "cluster_network_interface": "", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zlib_isal": "false", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zlib_level": "5", 2026-02-25T10:18:02.417 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zlib_winsize": "-15", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "compressor_zstd_level": "1", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crash_dir": "/var/lib/ceph/crash", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_allow_pg_split": "true", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_bluestore_cpu_set": "", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_bluestore_num_threads": "6", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_cpu_num": "0", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_cpu_set": "", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_osd_obc_lru_size": "512", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_osd_scheduler_concurrency": "0", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_osd_stat_interval": "0", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_poll_mode": "false", 2026-02-25T10:18:02.418 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_reactor_idle_poll_time_us": "200", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_reactor_io_latency_goal_ms": "0.000000", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "crimson_reactor_task_quota_ms": "0.500000", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "crush_location": "", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "crush_location_hook": "", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "crush_location_hook_timeout": "10", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "d4n_writecache_enabled": "false", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "daemonize": "false", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "daos_pool": "tank", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "dbstore_db_name_prefix": "dbstore", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "debug_alienstore": "0/5", 2026-02-25T10:18:02.419 INFO:teuthology.orchestra.run.trial068.stdout: "debug_allow_any_pool_priority": "false", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_asok": "1/5", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_asok_assert_abort": "false", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_asserts_on_shutdown": "true", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_auth": "20/20", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bdev": "1/3", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bluefs": "1/5", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bluestore": "1/5", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_bluestore_compression": "1/5", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_buffer": "0/1", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_ceph_dedup": "0/5", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_ceph_exporter": "1/5", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_cephfs_mirror": "0/5", 2026-02-25T10:18:02.420 INFO:teuthology.orchestra.run.trial068.stdout: "debug_cephsqlite": "0/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_client": "0/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_compressor": "1/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_context": "0/1", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_crimson_interrupt": "0/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_crush": "1/1", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_crypto": "1/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_cyanstore": "0/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_deliberately_leak_memory": "false", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_disable_randomized_ping": "false", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_dpdk": "1/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_eventtrace": "1/5", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_filer": "0/1", 2026-02-25T10:18:02.421 INFO:teuthology.orchestra.run.trial068.stdout: "debug_filestore": "1/3", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_finisher": "1/1", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_fuse": "1/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_heartbeat_testing_span": "0", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_heartbeatmap": "1/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_immutable_obj_cache": "0/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_javaclient": "1/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_journal": "1/3", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_journaler": "0/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_kstore": "1/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_lockdep": "0/1", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mclock": "1/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds": "1/5", 2026-02-25T10:18:02.422 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_balancer": "1/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_locker": "1/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_log": "1/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_log_expire": "1/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_migrator": "1/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mds_quiesce": "3/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_memstore": "1/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mgr": "2/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mgrc": "1/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_mon": "20/20", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_monc": "0/10", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_ms": "1/1", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_none": "0/5", 2026-02-25T10:18:02.423 INFO:teuthology.orchestra.run.trial068.stdout: "debug_objclass": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_objectcacher": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_objecter": "0/1", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_optracker": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_osd": "1/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_paxos": "20/20", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_perfcounter": "1/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_prioritycache": "1/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rados": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd_mirror": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd_pwl": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rbd_replay": "0/5", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_refs": "0/0", 2026-02-25T10:18:02.424 INFO:teuthology.orchestra.run.trial068.stdout: "debug_reserver": "1/1", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_access": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_bucket_logging": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_datacache": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_dbstore": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_dedup": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_flight": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_lifecycle": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_notification": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_restore": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rgw_sync": "1/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_rocksdb": "4/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore": "0/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_backref": "0/5", 2026-02-25T10:18:02.425 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_cache": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_cleaner": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_device": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_epm": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_fixedkv_tree": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_journal": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_lba": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_odata": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_omap": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_onode": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_t": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_seastore_tm": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_striper": "0/1", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_test": "0/5", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_throttle": "1/1", 2026-02-25T10:18:02.426 INFO:teuthology.orchestra.run.trial068.stdout: "debug_timer": "0/1", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "debug_tp": "0/5", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "debug_trace": "1/5", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "defer_client_eviction_on_laggy_osds": "false", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "device_failure_prediction_mode": "none", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "ec_extent_cache_size": "10485760", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "ec_pdw_write_mode": "0", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "enable_availability_tracking": "false", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_graylog": "false", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_journald": "false", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_stderr": "true", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "err_to_syslog": "false", 2026-02-25T10:18:02.427 INFO:teuthology.orchestra.run.trial068.stdout: "event_tracing": "false", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_addr": "0.0.0.0", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_cert_file": "", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_http_port": "9926", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_key_file": "", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_prio_limit": "5", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_sock_dir": "/var/run/ceph/", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_sort_metrics": "true", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "exporter_stats_period": "5", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "fake_statfs_for_testing": "0", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "fatal_signal_handlers": "true", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "filer_max_purge_ops": "10", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "filer_max_truncate_ops": "128", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_apply_finisher_threads": "1", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_blackhole": "false", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_btrfs_clone_range": "true", 2026-02-25T10:18:02.428 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_btrfs_snap": "true", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_caller_concurrency": "10", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_collect_device_partition_information": "true", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_commit_timeout": "600.000000", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_debug_inject_read_err": "false", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_debug_omap_check": "false", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_debug_verify_split": "false", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_dump_file": "", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_expected_throughput_ops": "200.000000", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fadvise": "true", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fail_eio": "true", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fd_cache_shards": "16", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fd_cache_size": "128", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fiemap": "false", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fiemap_threshold": "4096", 2026-02-25T10:18:02.429 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_fsync_flushes_journal_data": "false", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_index_retry_probability": "0.000000", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_inject_stall": "0", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_journal_parallel": "false", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_journal_trailing": "false", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_journal_writeahead": "false", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_kill_at": "0", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_alloc_hint_size": "1048576", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size": "0", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size_other": "512", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs": "0", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs_other": "2", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_inline_xattrs_xfs": "10", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_sync_interval": "5.000000", 2026-02-25T10:18:02.430 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size": "0", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size_other": "1024", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_merge_threshold": "-10", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_min_sync_interval": "0.010000", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_odsync_write": "false", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_omap_backend": "rocksdb", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_omap_backend_path": "", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_omap_header_cache_size": "1024", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_ondisk_finisher_threads": "1", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_op_thread_suicide_timeout": "180", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_op_thread_timeout": "60", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_op_threads": "2", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_punch_hole": "false", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2026-02-25T10:18:02.431 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_high_threshhold": "0.900000", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_low_threshhold": "0.300000", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_bytes": "104857600", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_queue_max_ops": "50", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_seek_data_hole": "false", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_sloppy_crc": "false", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_sloppy_crc_block_size": "65536", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_splice": "false", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_split_multiple": "2", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_split_rand_factor": "20", 2026-02-25T10:18:02.432 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_update_to": "1000", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_enable": "true", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_xfs_extsize": "false", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "filestore_zfs_snap": "false", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "fio_dir": "/tmp/fio", 2026-02-25T10:18:02.433 INFO:teuthology.orchestra.run.trial068.stdout: "fsid": "e959f599-67ec-4a01-98b7-0bdf9a6b1fe1", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_allow_other": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_atomic_o_trunc": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_big_writes": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_debug": "false", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_default_permissions": "false", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_disable_pagecache": "false", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_max_write": "0", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_multithreaded": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_require_active_mds": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_set_user_groups": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_splice_move": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_splice_read": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_splice_write": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_syncfs_on_mksnap": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "fuse_use_invalidate_cb": "true", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "gss_target_name": "ceph", 2026-02-25T10:18:02.434 INFO:teuthology.orchestra.run.trial068.stdout: "heartbeat_file": "", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "heartbeat_inject_failure": "0", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "heartbeat_interval": "5", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "host": "trial068", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_max_inflight_ops": "128", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_max_size": "1073741824", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_bps_burst": "0", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_bps_limit": "0", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_iops_burst": "0", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_iops_limit": "0", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "immutable_object_cache_watermark": "0.900000", 2026-02-25T10:18:02.435 INFO:teuthology.orchestra.run.trial068.stdout: "inject_early_sigterm": "false", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "jaeger_agent_port": "6799", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "jaeger_tracing_enable": "false", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_aio": "true", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_align_min_size": "65536", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_block_align": "true", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_block_size": "4096", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_dio": "true", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_discard": "false", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_force_aio": "false", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_ignore_corruption": "false", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_max_write_bytes": "10485760", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_max_write_entries": "100", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_replay_from": "0", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_high_multiple": "0.000000", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_high_threshhold": "0.900000", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_low_threshhold": "0.600000", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_throttle_max_multiple": "0.000000", 2026-02-25T10:18:02.436 INFO:teuthology.orchestra.run.trial068.stdout: "journal_write_header_frequency": "0", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "journal_zero_on_create": "false", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "journaler_prefetch_periods": "10", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "journaler_prezero_periods": "5", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "journaler_write_head_interval": "15", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "key": "", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "keyfile": "", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "librados_thread_count": "2", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "lockdep": "false", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "lockdep_force_backtrace": "false", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_coarse_timestamps": "true", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_flush_on_exit": "false", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_graylog_host": "127.0.0.1", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_graylog_port": "12201", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_max_new": "1000", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_max_recent": "10000", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_stderr_prefix": "", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_stop_at_utilization": "0.970000", 2026-02-25T10:18:02.437 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_file": "true", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_graylog": "false", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_journald": "false", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_stderr": "false", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "log_to_syslog": "false", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "max_rotating_auth_attempts": "10", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_action_on_write_error": "1", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_allow_async_dirops": "true", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_allow_batched_ops": "true", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_alternate_name_max": "8192", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_asio_thread_count": "2", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_export_pin": "true", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_dirs": "true", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_interval": "5", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_fragment_size_max": "100000", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_idle_threshold": "0.000000", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_interval": "10", 2026-02-25T10:18:02.438 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_max": "-1", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_max_until": "-1", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_merge_size": "50", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_midchunk": "0.300000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_min_rebalance": "0.100000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_min_start": "0.200000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_minchunk": "0.001000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_mode": "0", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_need_max": "1.200000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_need_min": "0.800000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_overload_epochs": "2", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_replicate_threshold": "8000.000000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_sample_interval": "3.000000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_bits": "3", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_rd": "25000.000000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_size": "10000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_split_wr": "10000.000000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_target_decay": "10.000000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_beacon_grace": "15.000000", 2026-02-25T10:18:02.439 INFO:teuthology.orchestra.run.trial068.stdout: "mds_beacon_interval": "4.000000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_beacon_mon_down_grace": "60", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_memory_limit": "4294967296", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_mid": "0.700000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_delay": "0", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_sleep": "200", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_splitauth": "true", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_quiesce_threshold": "524288", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_release_free_interval": "10", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_reservation": "0.050000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_trim_decay_rate": "1.000000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_trim_interval": "1", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cache_trim_threshold": "262144", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_client_delegate_inos_pct": "50", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_client_prealloc_inos": "1000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_connect_bootstrapping": "false", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_damage_table_max_entries": "10000", 2026-02-25T10:18:02.440 INFO:teuthology.orchestra.run.trial068.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_auth_pins": "false", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_frag": "false", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_scatterstat": "false", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_debug_subtrees": "false", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_decay_halflife": "5.000000", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_default_dir_hash": "2", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_defer_session_stale": "true", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_delay_journal_replay_for_testing": "0", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_deny_all_reconnect": "false", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_keys_per_op": "16384", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_max_commit_size": "10", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_max_entries": "0", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dir_prefetch": "true", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dirstat_min_interval": "1.000000", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_after_rejoin": "false", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_on_map": "false", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_threshold_file": "0", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_early_reply": "true", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_enable_op_tracker": "true", 2026-02-25T10:18:02.441 INFO:teuthology.orchestra.run.trial068.stdout: "mds_enforce_unique_name": "true", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_distributed": "true", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_random": "true", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_export_ephemeral_random_max": "0.010000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_extraordinary_events_dump_interval": "0", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_forward_all_requests_to_auth": "false", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_freeze_tree_timeout": "30.000000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_fscrypt_last_block_max_size": "4096", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_go_bad_corrupt_dentry": "true", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_health_cache_threshold": "1.500000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_health_summarize_threshold": "10", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_heartbeat_grace": "15.000000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_heartbeat_reset_grace": "1000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_health_dummy": "false", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_migrator_session_race": "false", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_skip_replaying_inotable": "false", 2026-02-25T10:18:02.442 INFO:teuthology.orchestra.run.trial068.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_join_fs": "", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_journal_format": "1", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_after_journal_logs_flushed": "false", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_create_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_dirfrag_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_export_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_import_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_journal_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_journal_expire_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_journal_replay_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_link_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_mdstable_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_openc_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_rename_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_kill_shutdown_at": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_event_large_threshold": "524288", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_events_per_segment": "1024", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_max_events": "-1", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_max_segments": "128", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_minor_segments_per_major_segment": "16", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_pause": "false", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_segment_size": "0", 2026-02-25T10:18:02.443 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_skip_corrupt_events": "false", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_skip_unbounded_events": "false", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_trim_decay_rate": "1.000000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_trim_threshold": "128", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_trim_upkeep_interval": "1000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_log_warn_factor": "2.000000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_caps_per_client": "1048576", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_completed_flushes": "100000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_completed_requests": "100000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_export_size": "20971520", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_file_recover": "32", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_purge_files": "64", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_purge_ops": "8192", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_scrub_ops_in_progress": "5", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_max_snaps_per_dir": "100", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_metrics_update_interval": "2", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_min_caps_per_client": "100", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_min_caps_working_set": "10000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_mon_shutdown_timeout": "5.000000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_numa_node": "-1", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_oft_prefetch_dirfrags": "false", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_complaint_time": "30.000000", 2026-02-25T10:18:02.444 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_duration": "600", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_size": "20", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_slow_op_size": "20", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_history_slow_op_threshold": "10", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_op_log_threshold": "5", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_ping_grace": "15", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_ping_interval": "5", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_global_max_decay_threshold": "131072", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_max_caps": "30000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_max_decay_rate": "1.500000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_max_decay_threshold": "131072", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_warning_decay_rate": "60.000000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_recall_warning_threshold": "262144", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_reconnect_timeout": "45.000000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_replay_interval": "1.000000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_replay_unsafe_with_closed_session": "false", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_request_load_average_decay_rate": "60.000000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_root_ino_gid": "0", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_root_ino_uid": "0", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_scatter_nudge_interval": "5.000000", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_scrub_stats_review_period": "1", 2026-02-25T10:18:02.445 INFO:teuthology.orchestra.run.trial068.stdout: "mds_server_dispatch_client_request_delay": "0", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_blocklist_on_evict": "true", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_blocklist_on_timeout": "true", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cache_liveness_magnitude": "10", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_cap_acquisition_throttle": "100000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_session_metadata_threshold": "16777216", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_sessionmap_keys_per_op": "1024", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_shutdown_check": "0", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_skip_ino": "0", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_sleep_rank_change": "0.000000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_snap_max_uid": "4294967294", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_snap_min_uid": "0", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_snap_rstat": "false", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_standby_replay_damaged": "false", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_symlink_recovery": "true", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_task_status_update_interval": "2.000000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_thrash_exports": "0", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_thrash_fragments": "0", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_tick_interval": "5.000000", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_use_global_snaprealm_seq_for_subvol": "true", 2026-02-25T10:18:02.446 INFO:teuthology.orchestra.run.trial068.stdout: "mds_valgrind_exit": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mds_verify_backtrace": "1", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mds_verify_scatter": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mds_wipe_ino_prealloc": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mds_wipe_sessions": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mempool_debug": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_debug_omit_block_device_write": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_device_bytes": "1073741824", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_page_set": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "memstore_page_size": "65536", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_client_bytes": "134217728", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_client_messages": "512", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_connect_retry_interval": "1.000000", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_disabled_modules": "", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_enable_op_tracker": "true", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_initial_modules": "iostat nfs", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_max_pg_creating": "1024", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_max_pg_num_change": "128", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mds_bytes": "134217728", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mds_messages": "128", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2026-02-25T10:18:02.447 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mon_bytes": "134217728", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_mon_messages": "128", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_num_op_tracker_shard": "32", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_complaint_time": "30.000000", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_duration": "600", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_size": "20", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_slow_op_size": "20", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_op_log_threshold": "5", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_osd_bytes": "536870912", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_osd_messages": "8192", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_pool": "true", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_service_beacon_grace": "60.000000", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_standby_modules": "true", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_stats_period": "5", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_stats_threshold": "5", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_tick_period": "2", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mgr_ttl_cache_expire_seconds": "0", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mon_accept_timeout_factor": "2.000000", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mon_allow_pool_delete": "true", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mon_allow_pool_size_one": "true", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mon_auth_validate_all_caps": "true", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2026-02-25T10:18:02.448 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_bytes": "104857600", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_directed_command_retry": "2", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval": "3.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_on_resend": "true", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_hunt_parallel": "3", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_log_interval": "1.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_max_log_entries_per_message": "1000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_ping_interval": "10.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_ping_timeout": "30.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_client_target_rank": "-1", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_clock_drift_allowed": "1.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_level": "debug", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_file": "true", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_graylog": "false", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_graylog_port": "12201", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_journald": "false", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_stderr": "false", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_syslog": "default=false", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2026-02-25T10:18:02.449 INFO:teuthology.orchestra.run.trial068.stdout: "mon_compact_on_bootstrap": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_compact_on_start": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_compact_on_trim": "true", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_con_tracker_persist_interval": "10", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_con_tracker_score_halflife": "43200", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_config_key_max_entry_size": "65536", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_cpu_threads": "4", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_crush_min_required_version": "hammer", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_daemon_bytes": "419430400", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data_avail_crit": "5", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data_avail_warn": "5", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_data_size_warn": "16106127360", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_block_osdmap_trim": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_deprecated_as_obsolete": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_dump_json": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_dump_transactions": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_extra_checks": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_initial_persistent_features": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_require_squid": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_no_require_tentacle": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_delta_reset_interval": "10.000000", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_dns_srv_name": "ceph-mon", 2026-02-25T10:18:02.450 INFO:teuthology.orchestra.run.trial068.stdout: "mon_down_added_grace": "180", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_down_mkfs_grace": "120", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_down_uptime_grace": "60", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_election_default_strategy": "1", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_election_timeout": "5.000000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_elector_ping_divisor": "2", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_elector_ping_timeout": "2.000000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_enable_op_tracker": "true", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_fake_pool_delete": "false", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_force_quorum_join": "false", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_fsmap_prune_threshold": "300", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_globalid_prealloc": "10000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_detail_to_clog": "false", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_log_update_period": "5", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_max_detail": "50", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_to_clog": "true", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_to_clog_interval": "600", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_host": "10.20.193.68", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_host_override": "", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_initial_members": "", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_transaction_delay_max": "10.000000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2026-02-25T10:18:02.451 INFO:teuthology.orchestra.run.trial068.stdout: "mon_keyvaluedb": "rocksdb", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_lease": "5.000000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_lease_renew_interval_factor": "0.600000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_log_full_interval": "50", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_log_max": "10000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_log_max_summary": "50", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_log_entries_per_event": "4096", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_log_epochs": "500", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_mdsmap_epochs": "500", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_mgrmap_epochs": "500", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_nvmeof_epochs": "500", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_osd": "10000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_pg_per_osd": "10000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_pool_pg_num": "65536", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_max_snap_prune_per_epoch": "100", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mds_blocklist_interval": "86400.000000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mds_force_trim_to": "0", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mds_skip_sanity": "false", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_memory_autotune": "true", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_memory_target": "2147483648", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_beacon_grace": "30", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_digest_period": "5", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_inactive_grace": "60", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_mkfs_grace": "240", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_min_osdmap_epochs": "50", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_netsplit_grace_period": "9", 2026-02-25T10:18:02.452 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_beacon_grace": "7", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_beacons_till_ack": "15", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_delete_grace": "900", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_skip_failovers_interval": "16", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_nvmeofgw_wrong_map_ignore_sec": "15", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_complaint_time": "30", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_duration": "600", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_size": "20", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_slow_op_size": "20", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_history_slow_op_threshold": "10", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_op_log_threshold": "5", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_adjust_down_out_interval": "true", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_auto_mark_in": "false", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_auto_mark_new_in": "true", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_cache_size": "500", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_cache_size_min": "134217728", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_crush_smoke_test": "true", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_destroyed_out_interval": "600", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_down_out_interval": "600", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_down_out_subtree_limit": "rack", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_force_trim_to": "0", 2026-02-25T10:18:02.453 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_full_ratio": "0.950000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_laggy_halflife": "3600", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_laggy_max_interval": "300", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_laggy_weight": "0.300000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_max_initial_pgs": "1024", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_min_down_reporters": "2", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_min_in_ratio": "0.750000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_min_up_ratio": "0.300000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_nearfull_ratio": "0.850000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_prime_pg_temp": "true", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_report_timeout": "900", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_reporter_subtree_level": "osd", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_warn_num_repaired": "10", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osd_warn_op_age": "32.000000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_enabled": "true", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_interval": "2", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_min": "15", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_osdmap_full_prune_txsize": "2", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_stuck_threshold": "60", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_min_objects": "10000", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_min_per_osd": "0", 2026-02-25T10:18:02.454 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pg_warn_min_pool_objects": "1000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pool_quota_crit_threshold": "0", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_pool_quota_warn_threshold": "0", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_probe_timeout": "2.000000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_max_change": "0.050000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_max_osds": "4", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_min_bytes_per_osd": "10", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_reweight_min_pgs_per_osd": "4", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_interval": "86400", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_max_keys": "100", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_scrub_timeout": "300", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_session_timeout": "300", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_smart_report_timeout": "5", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stat_smooth_intervals": "6", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_pool_min_size": "2", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_pool_size": "4", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_subscribe_interval": "86400.000000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_debug": "false", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_max_payload_keys": "2000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_max_payload_size": "1048576", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_provider_kill_at": "0", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_requester_kill_at": "0", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_sync_timeout": "60.000000", 2026-02-25T10:18:02.455 INFO:teuthology.orchestra.run.trial068.stdout: "mon_target_pg_per_osd": "100", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_tick_interval": "5", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_timecheck_interval": "300.000000", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_timecheck_skew_interval": "30.000000", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_use_min_delay_socket": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_older_version_delay": "604800", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_filestore_osds": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_misplaced": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_older_version": "true", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_no_app": "true", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_no_app_grace": "300", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_no_redundancy": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_on_too_few_osds": "false", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "monmap": "", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2026-02-25T10:18:02.456 INFO:teuthology.orchestra.run.trial068.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "motr_tracing_enabled": "false", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_op_threads": "3", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_buffer_size": "131072", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_cm": "false", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_device_name": "", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_dscp": "96", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_enable_hugepage": "false", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_gid_idx": "0", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_local_gid": "", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_polling_us": "1000", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_port_num": "1", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_receive_buffers": "32768", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_receive_queue_len": "4096", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_roce_ver": "1", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_send_buffers": "1024", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_sl": "3", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_support_srq": "true", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_rdma_type": "ib", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_async_reap_threshold": "5", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_before_connect": "false", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_ipv4": "true", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_ipv6": "false", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_msgr1": "true", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_msgr2": "true", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_port_max": "7568", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_port_min": "6800", 2026-02-25T10:18:02.457 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_prefer_ipv4": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_retry_count": "3", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_bind_retry_delay": "5", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_client": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_mds": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_mgr": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_mon": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_blackhole_osd": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_client_mode": "crc", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_client_throttle_retry_time_interval": "5000", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_cluster_mode": "crc", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_cluster_type": "", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_compress_secure": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_connection_idle_timeout": "900", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_connection_ready_timeout": "10", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_crc_data": "true", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_crc_header": "true", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_bad_msg": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_bug": "true", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_old_message": "true", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_skipped_message": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_die_on_unhandled_msg": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dispatch_throttle_bytes": "104857600", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_coremask": "0xF", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_debug_allow_loopback": "false", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_devs_allowlist": "", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_enable_tso": "true", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_host_ipv4_addr": "", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_hugepages": "", 2026-02-25T10:18:02.458 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_hw_flow_control": "true", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_lro": "true", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_memory_channel": "4", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_pmd": "", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_port_id": "0", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dump_corrupt_message_level": "1", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_dump_on_send": "false", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_initial_backoff": "0.200000", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_delay_max": "1.000000", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_delay_probability": "0.000000", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_delay_type": "", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_internal_delays": "0.000000", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_network_congestion": "0", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_inject_socket_failures": "0", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_learn_addr_from_peer": "true", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_max_accept_failures": "4", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_max_backoff": "15.000000", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_mon_client_mode": "crc", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_mon_cluster_mode": "crc", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_mon_service_mode": "crc", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_osd_compress_min_size": "1024", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_osd_compress_mode": "none", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_osd_compression_algorithm": "snappy", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_pq_min_cost": "65536", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_public_type": "", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_service_mode": "crc", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_listen_backlog": "512", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_nodelay": "true", 2026-02-25T10:18:02.459 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_prefetch_max_size": "65536", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "ms_tcp_rcvbuf": "0", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "ms_time_events_min_wait_interval": "1000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "ms_type": "async+posix", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "no_config_file": "false", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "nvmeof_mon_client_connect_panic": "30", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "nvmeof_mon_client_tick_period": "1", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_completion_locks_per_session": "32", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_debug_inject_relock_delay": "false", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_inflight_op_bytes": "104857600", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_inflight_ops": "1024", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_inject_no_watch_ping": "false", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_retry_writes_after_first_reply": "false", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_tick_interval": "5.000000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objecter_timeout": "10.000000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objectstore_blackhole": "false", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "openssl_conf": "", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_delay_time": "5.000000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_hist_halflife": "1000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_max_low_ops": "2", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_max_ops": "4", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_min_evict_effort": "0.100000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_quantize_effort": "0.100000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_agent_slop": "0.020000", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_aggregated_slow_ops_logging": "true", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_allow_recovery_below_min_size": "true", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_asio_thread_count": "2", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_async_recovery_min_cost": "100", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_auto_mark_unfound_lost": "false", 2026-02-25T10:18:02.460 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backfill_retry_interval": "30.000000", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backfill_scan_max": "512", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backfill_scan_min": "64", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backoff_on_degraded": "false", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backoff_on_peering": "false", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_backoff_on_unfound": "true", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_beacon_report_interval": "300", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_duration": "30", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_large_size_max_throughput": "104857600", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_max_block_size": "67108864", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_bench_small_size_max_iops": "100", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_blkin_trace_all": "false", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_blocked_scrub_grace_period": "120", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_check_for_log_corruption": "false", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_check_max_object_name_len_on_startup": "true", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_class_update_on_start": "true", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_message_cap": "256", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_message_size_cap": "524288000", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_op_priority": "63", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_client_watch_timeout": "30", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_command_max_records": "256", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_compact_on_start": "false", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_copyfrom_max_chunk": "8388608", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_chooseleaf_type": "0", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_initial_weight": "-1.000000", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_update_on_start": "true", 2026-02-25T10:18:02.461 INFO:teuthology.orchestra.run.trial068.stdout: "osd_crush_update_weight_set": "true", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_drop_ping_duration": "0", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_drop_ping_probability": "0.000000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_feed_pullee": "-1", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_inject_copyfrom_error": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_misdirected_ops": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_no_acting_change": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_no_purge_strays": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_op_order": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_pg_log_writeout": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_pretend_recovery_active": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_random_push_read_error": "0.000000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_shutdown": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_trim_objects": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_cached_snaps": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_missing_on_start": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_snaps": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_debug_verify_stray_on_activate": "false", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_interval": "604800.000000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_keys": "1024", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2026-02-25T10:18:02.462 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_stride": "4194304", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_default_data_pool_replay_window": "5", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_default_notify_timeout": "30", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep": "0.000000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep_hdd": "5.000000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep_hybrid": "1.000000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_delete_sleep_ssd": "1.000000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_discard_disconnected_ops": "true", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_ec_partial_reads": "true", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_enable_op_tracker": "true", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_extblkdev_plugins": "vdo", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_failsafe_full_ratio": "0.970000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_fail_on_connection_refused": "true", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_info": "true", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_shutdown": "true", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_shutdown_notify_mon": "true", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_fast_shutdown_timeout": "15", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_find_best_info_ignore_history_les": "false", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_force_auth_primary_missing_objects": "100", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_function_tracing": "false", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_grace": "20", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_interval": "6", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_min_peers": "10", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_min_size": "2000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_stale": "600", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_hit_set_max_size": "100000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_hit_set_min_size": "1000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_hit_set_namespace": ".ceph-internal", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_ignore_stale_divergent_priors": "false", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2026-02-25T10:18:02.463 INFO:teuthology.orchestra.run.trial068.stdout: "osd_inject_failure_on_pg_removal": "false", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_journal_flush_on_shutdown": "true", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_journal_size": "5120", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_kill_backfill_at": "0", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_loop_before_reset_tphandle": "64", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_cache_size": "50", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_dedup": "true", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_message_max": "40", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_message_max_bytes": "10485760", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_map_share_max_epochs": "40", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_attr_name_len": "100", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_attr_size": "0", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_backfills": "1", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_markdown_count": "5", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_markdown_period": "600", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_object_name_len": "2048", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_object_namespace_len": "256", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_object_size": "134217728", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_omap_entries_per_request": "1024", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pg_blocked_by": "16", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pg_log_entries": "2", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_pgls": "1024", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_push_cost": "8388608", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_push_objects": "10", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_scrubs": "3", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_trimming_pgs": "2", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_write_op_reply_len": "64", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_max_write_size": "90", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_low_threshold_hdd": "50.000000", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_low_threshold_ssd": "1000.000000", 2026-02-25T10:18:02.464 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_override_recovery_settings": "false", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_profile": "balanced", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_scheduler_client_wgt": "1", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mclock_skip_benchmark": "false", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_base": "805306368", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_cache_min": "134217728", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_cache_resize_interval": "1.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_expected_fragmentation": "0.150000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_target": "4294967296", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_target_autotune": "false", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_min_pg_log_entries": "1", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_min_recovery_priority": "0", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_heartbeat_interval": "30", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_report_interval": "5", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_report_max_in_flight": "2", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_mon_shutdown_timeout": "5.000000", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_num_cache_shards": "32", 2026-02-25T10:18:02.465 INFO:teuthology.orchestra.run.trial068.stdout: "osd_num_op_tracker_shard": "32", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_numa_auto_affinity": "true", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_numa_node": "-1", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_numa_prefer_iface": "true", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_object_clean_region_max_num_intervals": "10", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objecter_finishers": "1", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore": "bluestore", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore_fuse": "false", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore_ideal_list_max": "64", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_objectstore_tracing": "false", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_complaint_time": "30.000000", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_duration": "600", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_size": "20", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_slow_op_size": "20", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_log_threshold": "5", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_shards": "0", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_shards_hdd": "1", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_shards_ssd": "8", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_threads_per_shard": "0", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_threads_per_shard_hdd": "5", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_pq_min_cost": "65536", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_queue": "mclock_scheduler", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_queue_cut_off": "high", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_thread_suicide_timeout": "150", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_op_thread_timeout": "15", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_open_classes_on_start": "true", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_os_flags": "0", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_peering_op_priority": "255", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_delete_cost": "1048576", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_delete_priority": "5", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2026-02-25T10:18:02.466 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_log_dups_tracked": "3000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_log_trim_max": "10000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_log_trim_min": "0", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_object_context_cache_count": "64", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_min_evict_age": "0", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_min_flush_age": "0", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_crimson": "true", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_crush_rule": "-1", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_ec_fast_read": "false", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_erasure_code_profile": "plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_bulk": "false", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_ec_optimizations": "false", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_hashpspool": "true", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_nodelete": "false", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_nopgchange": "false", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flag_nosizechange": "false", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_flags": "0", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_min_size": "2", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_pg_num": "32", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_pgp_num": "0", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_read_ratio": "70", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_size": "3", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_default_type": "replicated", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_erasure_code_stripe_unit": "0", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_pool_use_gmt_hitset": "true", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_push_per_object_cost": "1000", 2026-02-25T10:18:02.467 INFO:teuthology.orchestra.run.trial068.stdout: "osd_read_ec_check_for_errors": "false", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recover_clone_overlap": "true", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recover_clone_overlap_limit": "10", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_cost": "20971520", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_delay_start": "0.000000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_active": "0", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_active_hdd": "3", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_active_ssd": "10", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_chunk": "8388608", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_max_single_start": "1", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_op_priority": "3", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_op_warn_multiple": "16", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_priority": "5", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_retry_interval": "30.000000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep": "0.000000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded": "0.000000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded_hdd": "0.100000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded_hybrid": "0.025000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_degraded_ssd": "0.000000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_hdd": "0.100000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_recovery_sleep_ssd": "0.000000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_requested_scrub_priority": "5", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_rollback_to_cluster_snap": "", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_auto_repair": "false", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_auto_repair_num_errors": "5", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_backoff_ratio": "0.660000", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_begin_hour": "0", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_begin_week_day": "0", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_chunk_max": "15", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_chunk_min": "5", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_cost": "52428800", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_disable_reservation_queuing": "false", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_during_recovery": "false", 2026-02-25T10:18:02.468 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_end_hour": "0", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_end_week_day": "0", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_event_cost": "4096", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_extended_sleep": "0.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_invalid_stats": "true", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_load_threshold": "10.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_max_interval": "604800.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_max_preemptions": "5", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_min_interval": "86400.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_priority": "5", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_after_noscrub": "60", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_delay": "30", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_new_interval": "10", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_pg_state": "60", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_retry_trimming": "10", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_scrub_sleep": "0.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_shallow_scrub_chunk_max": "100", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_shallow_scrub_chunk_min": "50", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_shutdown_pgref_assert": "false", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_skip_check_past_interval_bounds": "false", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_skip_data_digest": "false", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_smart_report_timeout": "5", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_cost": "1048576", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_priority": "5", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep": "0.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_stats_update_period_not_scrubbing": "120", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_stats_update_period_scrubbing": "15", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_target_transaction_size": "30", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_count": "4", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2026-02-25T10:18:02.469 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_default_cache_mode": "writeback", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tier_promote_max_objects_sec": "25", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osd_tracing": "false", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osd_use_stale_snap": "false", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "osdc_blkin_trace_all": "false", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_kill_at": "0", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_max_join_drift": "10", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_min": "500", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_min_wait": "0.050000", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_propose_interval": "1.000000", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_service_trim_max": "500", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_service_trim_max_multiplier": "20", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_service_trim_min": "10", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_stash_full_interval": "25", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_trim_max": "500", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "paxos_trim_min": "250", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "perf": "true", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "plugin_crypto_accelerator": "crypto_isal", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "plugin_dir": "/usr/lib64/ceph", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "pool_availability_update_interval": "1.000000", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "public_addr": "-", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "public_addrv": "", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "public_bind_addr": "-", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "public_network": "", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "public_network_interface": "", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "qat_compressor_busy_polling": "false", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "qat_compressor_enabled": "false", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "qat_compressor_session_max_number": "256", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "rados_mon_op_timeout": "0", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "rados_osd_op_timeout": "0", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "rados_replica_read_policy": "default", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "rados_replica_read_policy_on_objclass": "false", 2026-02-25T10:18:02.470 INFO:teuthology.orchestra.run.trial068.stdout: "rados_tracing": "false", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_atime_update_interval": "60", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_balance_parent_reads": "false", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_balance_snap_reads": "false", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_blkin_trace_all": "false", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_blocklist_expire_seconds": "0", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_blocklist_on_break_lock": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_block_writes_upfront": "false", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_max_dirty": "25165824", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_max_dirty_age": "1.000000", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_max_dirty_object": "0", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_policy": "writearound", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_size": "33554432", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_target_dirty": "16777216", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_cache_writethrough_until_flush": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_clone_copy_on_read": "false", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_compression_hint": "none", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_concurrent_management_ops": "10", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_config_pool_override_update_timestamp": "0", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_clone_format": "auto", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_data_pool": "", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_features": "61", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_format": "2", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_map_options": "", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_order": "22", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_pool": "rbd", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_stripe_count": "0", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_default_stripe_unit": "0", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_disable_zero_copy_writes": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_discard_granularity_bytes": "65536", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_discard_on_zeroed_write_same": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_enable_alloc_hint": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_io_scheduler": "simple", 2026-02-25T10:18:02.471 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_commit_age": "5.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_max_payload_bytes": "16384", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_flush_age": "0.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_flush_bytes": "1048576", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_flush_interval": "0", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_order": "24", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_pool": "", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_journal_splay_width": "4", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_localize_parent_reads": "false", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_localize_snap_reads": "false", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_type": "simple", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_image_state_check_interval": "30", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_autotune": "true", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_base": "805306368", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_cache_min": "134217728", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_memory_target": "4294967296", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_perf_stats_prio": "5", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2026-02-25T10:18:02.472 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_delete_delay": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_replay_delay": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_move_to_trash_on_remove": "false", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_mtime_update_interval": "60", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_non_blocking_aio": "true", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_op_thread_timeout": "60", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_op_threads": "1", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_parent_cache_enabled": "false", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_persistent_cache_mode": "disabled", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_persistent_cache_path": "/tmp", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_persistent_cache_size": "1073741824", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_plugins": "", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_bps_burst": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_bps_burst_seconds": "1", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_bps_limit": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_exclude_ops": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_iops_burst": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_iops_burst_seconds": "1", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_iops_limit": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_bps_burst": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_bps_limit": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_iops_burst": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_read_iops_limit": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_schedule_tick_min": "50", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_bps_burst": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_bps_limit": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_iops_burst": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_qos_write_iops_limit": "0", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_quiesce_notification_attempts": "10", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_read_from_replica_policy": "default", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_readahead_max_bytes": "524288", 2026-02-25T10:18:02.473 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_readahead_trigger_requests": "10", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_request_timed_out_seconds": "30", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_skip_partial_discard": "true", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_tracing": "false", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_validate_names": "true", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rbd_validate_pool": "true", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "restapi_base_url": "", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "restapi_log_level": "", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_account_default_quota_max_objects": "-1", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_account_default_quota_max_size": "-1", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_acl_grants_max_num": "100", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_admin_entry": "admin", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_asio_assert_yielding": "false", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_backend_store": "rados", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_barbican_url": "", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_beast_enable_async": "true", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_counters_cache": "false", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_counters_cache_size": "10000", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_default_quota_max_size": "-1", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_eexist_override": "false", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_index_max_aio": "128", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_logging_obj_roll_time": "300", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_persistent_notif_num_shards": "11", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_quota_cache_size": "10000", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_quota_ttl": "600", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_bucket_sync_spawn_window": "20", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cache_enabled": "true", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cache_expiry_interval": "900", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cache_lru_size": "25000", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_config_store": "rados", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_content_length_compat": "false", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_copy_obj_progress": "true", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cors_rules_max_num": "100", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_cross_domain_policy": "", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_default_encryption_key": "", 2026-02-25T10:18:02.474 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_addr": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_ca_path": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_client_cert": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_client_key": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_kms_key_template": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_password": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_kmip_username": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_require_ssl": "true", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_backend": "vault", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_suppress_logs": "true", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_addr": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_auth": "token", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_namespace": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_prefix": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_secret_engine": "transit", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_ssl_cacert": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_token_file": "", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_crypt_vault_verify_ssl": "true", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_buffersize": "524288", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_low_speed_limit": "1024", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_low_speed_time": "300", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_tcp_keepalive": "0", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_curl_wait_timeout_ms": "1000", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2026-02-25T10:18:02.475 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_fadvise": "4", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_libaio_aio_num": "64", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d3n_libaio_aio_threads": "20", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_backend_address": "127.0.0.1:6379", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_cache_cleaning_interval": "1000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_datacache_disk_reserve": "1073741824", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_fadvise": "4", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_l1_write_open_flags": "4096", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_libaio_aio_num": "64", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_libaio_aio_threads": "20", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_d4n_local_rgw_address": "127.0.0.1:8000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_log_changes_size": "1000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_log_num_shards": "128", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_log_window": "30", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_notify_interval_msec": "0", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_sync_poll_interval": "20", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_data_sync_spawn_window": "20", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_debug_inject_set_olh_err": "0", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_data_log_backing": "fifo", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_realm_info_oid": "default.realm", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_region_info_oid": "default.region", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_zone_info_oid": "default.zone", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_defer_to_bucket_acls": "", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_delete_multi_obj_max_num": "1000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_disable_s3select": "false", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_admin_lim": "0.000000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_admin_res": "100.000000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_auth_lim": "0.000000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_auth_res": "200.000000", 2026-02-25T10:18:02.476 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_data_lim": "0.000000", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_data_res": "500.000000", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_data_wgt": "500.000000", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_metadata_res": "500.000000", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dns_name": "", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dns_s3website_name": "", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dynamic_resharding": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_gc_threads": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_jwks_url_verification": "false", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_lc_threads": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_mdsearch": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_ops_log": "false", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_quota_threads": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_restore_threads": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_static_website": "false", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enable_usage_log": "false", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_enforce_swift_acls": "true", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_exit_timeout_secs": "120", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_expose_bucket": "false", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_extended_http_attrs": "", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_filter": "none", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_frontends": "beast port=7480", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_concurrent_io": "10", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_deferred": "50", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_objs": "32", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_queue_size": "134213632", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_max_trim_chunk": "16", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_obj_min_wait": "7200", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_processor_max_time": "3600", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_gc_processor_period": "3600", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_get_obj_max_req_size": "4194304", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_get_obj_window_size": "16777216", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_graceful_stop": "false", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_healthcheck_disabling_path": "", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_http_notif_connection_timeout": "5", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_http_notif_max_inflight": "8192", 2026-02-25T10:18:02.477 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_http_notif_message_timeout": "10", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ignore_get_invalid_range": "false", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_init_timeout": "300", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_kafka_connection_idle": "300", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_kafka_message_timeout": "5000", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_kafka_sleep_timeout": "10", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_accepted_admin_roles": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_accepted_reader_roles": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_domain": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_password": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_password_path": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_project": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_tenant": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_admin_user": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_domain": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_password": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_project": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_tenant": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_barbican_user": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_implicit_tenants": "false", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_service_token_enabled": "false", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_token_cache_size": "10000", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_token_cache_ttl": "300", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_url": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_keystone_verify_ssl": "true", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_debug_interval": "-1", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_list_cnt": "1000", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_lock_max_time": "90", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_objs": "32", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_rules": "1000", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_worker": "3", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_max_wp_worker": "128", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_ordered_list_threshold": "500", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lc_thread_delay": "0", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_dnattr": "uid", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_searchfilter": "", 2026-02-25T10:18:02.478 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ldap_uri": "", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lfuda_sync_frequency": "60", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_list_bucket_min_readahead": "1000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_list_buckets_max_chunk": "1000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_http_headers": "", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_nonexistent_bucket": "false", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_log_object_name_utc": "false", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lua_enable": "true", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lua_max_memory_per_state": "128000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_lua_max_runtime_per_state": "1000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_attr_name_len": "0", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_attr_size": "0", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_attrs_num_in_req": "0", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_chunk_size": "4194304", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_concurrent_requests": "1024", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_control_aio": "8", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_dynamic_shards": "1999", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_listing_results": "5000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_notify_retries": "10", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_objs_per_shard": "100000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_put_param_size": "1048576", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_put_size": "5368709120", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_max_slo_entries": "1000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_md_log_max_shards": "64", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_md_notify_interval_msec": "200", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_meta_sync_poll_interval": "20", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_meta_sync_spawn_window": "20", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_mime_types_file": "/etc/mime.types", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_mp_lock_max_time": "600", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_multi_obj_del_max_aio": "16", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_multipart_min_part_size": "5242880", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_multipart_part_upload_limit": "10000", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_fhcache_partitions": "3", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_fhcache_size": "2017", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_frontends": "rgw-nfs", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_lru_lanes": "5", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_max_gc": "300", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_namespace_expire_secs": "300", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_gc_threads": "false", 2026-02-25T10:18:02.479 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_lc_threads": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_quota_threads": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_restore_threads": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_run_sync_thread": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_s3_fast_attrs": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_nfs_write_completion_interval_s": "10", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_num_async_rados_threads": "32", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_num_control_oids": "8", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_numa_node": "-1", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_obj_stripe_size": "4194304", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_obj_tombstone_cache_size": "1000", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_objexp_chunk_size": "100", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_objexp_gc_interval": "600", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_objexp_hints_num_shards": "127", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_olh_pending_timeout_sec": "3600", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_op_thread_suicide_timeout": "0", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_op_thread_timeout": "600", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_op_tracing": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_opa_token": "", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_opa_url": "", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_opa_verify_ssl": "true", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_data_backlog": "5242880", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_rados": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ops_log_socket_path": "", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_override_bucket_index_max_shards": "0", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_parquet_buffer_size": "16777216", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_push_interval": "2.000000", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_push_interval_max": "30.000000", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_period_root_pool": ".rgw.root", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_policy_reject_invalid_principals": "true", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_lanes": "3", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_lmdb_count": "3", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_max_buckets": "100", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_cache_partitions": "3", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_posix_userdb_dir": "/var/lib/ceph/radosgw", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_print_continue": "true", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_print_prohibited_content_length": "false", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_put_obj_max_window_size": "67108864", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_put_obj_min_window_size": "16777216", 2026-02-25T10:18:02.480 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rados_pool_recovery_priority": "5", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rados_tracing": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_ratelimit_interval": "60", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_realm": "", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_realm_id": "", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_realm_root_pool": ".rgw.root", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_redis_connection_pool_size": "512", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_region": "", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_region_root_pool": ".rgw.root", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_relaxed_region_enforcement": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_relaxed_s3_bucket_names": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_request_uri": "", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_batch_size": "64", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_bucket_lock_duration": "360", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_debug_interval": "-1", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_max_aio": "128", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_num_logs": "16", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_progress_judge_interval": "120", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshard_thread_interval": "600", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_reshardlog_threshold": "30000", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_resolve_cname": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_rest_getusage_op_compat": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_debug_interval": "-1", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_lock_max_time": "90", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_max_objs": "32", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_restore_processor_period": "900", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_run_sync_thread": "true", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_disable_signature_url": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_order": "sts, external, local", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_keystone": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_ldap": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_rados": "true", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_auth_use_sts": "false", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_client_max_sig_ver": "-1", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_s3_success_create_obj_status": "0", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_safe_max_objects_per_shard": "102400", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_scheduler_type": "throttler", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_script_uri": "", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_service_provider_name": "", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_shard_warning_threshold": "90.000000", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_client_id": "", 2026-02-25T10:18:02.481 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_client_secret": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_entry": "sts", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_key": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_max_session_duration": "43200", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_min_session_duration": "900", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sts_token_introspection_url": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_account_in_url": "false", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_auth_entry": "auth", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_auth_url": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_custom_header": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_enforce_content_length": "false", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_need_stats": "true", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_tenant_name": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_token_expiration": "86400", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_url": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_url_prefix": "swift", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_swift_versioning_enabled": "false", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_lease_period": "120", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_interval": "1200", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_max_buckets": "16", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_obj_etag_verify": "false", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_trace_history_size": "4096", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_trace_per_node_log_size": "32", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_thread_pool_size": "512", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_persistency_max_retries": "0", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_persistency_sleep_duration": "0", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_persistency_time_to_live": "0", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_topic_require_publish_policy": "false", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_comment": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_createby": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_encoding": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_flag": "false", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_max_size": "5368709120", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_origin": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_sha_unit": "524288", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_torrent_tracker": "", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_trust_forwarded_https": "false", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_log_flush_threshold": "1024", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_log_key_transition": "true", 2026-02-25T10:18:02.482 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_log_tick_interval": "30", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_max_shards": "32", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_usage_max_user_shards": "1", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_use_opa_authz": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_counters_cache": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_counters_cache_size": "10000", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_default_quota_max_objects": "-1", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_default_quota_max_size": "-1", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_max_buckets": "1000", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_policies_max_num": "100", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_sync_idle_users": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_sync_interval": "86400", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_quota_sync_wait_time": "86400", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_user_unique_email": "true", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_verify_ssl": "true", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_website_routing_rules_max_num": "50", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zone": "", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zone_id": "", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zone_root_pool": ".rgw.root", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zonegroup": "", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zonegroup_id": "", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgwlc_auto_session_clear": "true", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rgwlc_skip_bucket_step": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_block_size": "4096", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_bloom_bits_per_key": "20", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_row_ratio": "0.000000", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_shard_bits": "4", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_size": "536870912", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cache_type": "binned_lru", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cf_compact_on_deletion": "true", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_collect_compaction_stats": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_collect_extended_stats": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_collect_memory_stats": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_delete_range_threshold": "1048576", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_index_type": "binary_search", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_log_to_ceph_log": "true", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_metadata_block_size": "4096", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_partition_filters": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_perf": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2026-02-25T10:18:02.483 INFO:teuthology.orchestra.run.trial068.stdout: "rotating_keys_bootstrap_timeout": "30", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "rotating_keys_renewal_timeout": "10", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "run_dir": "/var/run/ceph", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_block_create": "true", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_2q_in_ratio": "0.500000", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_2q_out_ratio": "0.500000", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_size_pershard": "2147483648", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cachepin_type": "LRU", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cbjournal_size": "5368709120", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_cold_tier_generations": "3", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_data_delta_based_overwrite": "0", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_default_max_object_size": "16777216", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_default_object_metadata_reservation": "16777216", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_device_size": "53687091200", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_disable_end_to_end_data_protection": "true", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_full_integrity_check": "false", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_hot_tier_generations": "5", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_batch_capacity": "16", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_batch_flush_size": "16777216", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_journal_iodepth_limit": "5", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_main_device_type": "SSD", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_max_concurrent_transactions": "128", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_max_data_allocation_size": "0", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_segment_cleaner_gc_formula": "cost_benefit", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "seastore_segment_size": "67108864", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "service_unique_id": "", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "set_keepcaps": "false", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "setgroup": "", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "setuser": "", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "setuser_match_path": "", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "subv_metrics_window_interval": "30", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "target_max_misplaced_ratio": "0.050000", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "thp": "false", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "threadpool_default_timeout": "60", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "threadpool_empty_queue_max_wait": "2", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "throttler_perf_counter": "true", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "tmp_dir": "/tmp", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "uadk_compressor_enabled": "false", 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout: "uadk_wd_sync_ctx_num": "2" 2026-02-25T10:18:02.484 INFO:teuthology.orchestra.run.trial068.stdout:} 2026-02-25T10:18:02.486 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2026-02-25T10:18:02.561 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.558+0000 7f6b16006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.564 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.563+0000 7f6b16006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.573 INFO:teuthology.orchestra.run.trial068.stdout:{} 2026-02-25T10:18:02.573 INFO:teuthology.orchestra.run.trial068.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2026-02-25T10:18:02.580 DEBUG:teuthology.orchestra.run.trial068:> type ceph-objectstore-tool 2026-02-25T10:18:02.596 INFO:teuthology.orchestra.run.trial068.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T10:18:02.596 DEBUG:teuthology.orchestra.run.trial072:> type ceph-objectstore-tool 2026-02-25T10:18:02.612 INFO:teuthology.orchestra.run.trial072.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T10:18:02.613 INFO:teuthology.run_tasks:Running task rados... 2026-02-25T10:18:02.637 INFO:tasks.rados:Beginning rados... 2026-02-25T10:18:02.637 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 500, 'op_weights': {'write': 100}, 'ops': 4000, 'write_fadvise_dontneed': True} 2026-02-25T10:18:02.637 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T10:18:02.637 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 500, 'op_weights': {'write': 100}, 'ops': 4000, 'write_fadvise_dontneed': True} 2026-02-25T10:18:02.638 DEBUG:teuthology.run_tasks:Unwinding manager rados 2026-02-25T10:18:02.653 INFO:tasks.rados:joining rados 2026-02-25T10:18:02.653 INFO:tasks.thrashosds.thrasher:starting do_thrash 2026-02-25T10:18:02.653 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T10:18:02.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T10:18:02.654 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T10:18:02.654 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2026-02-25T10:18:02.655 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:18:02.655 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2026-02-25T10:18:02.655 INFO:tasks.rados:starting run 0 out of 1 2026-02-25T10:18:02.656 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-25T10:18:02.656 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-25T10:18:02.735 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.732+0000 7fd09828b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.738 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.733+0000 7fec4b7b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.738 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.736+0000 7fd09828b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.739 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.733+0000 7fb4b20cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.740 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.738+0000 7fec4b7b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.742 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:02.740+0000 7fb4b20cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:02.857 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-25T10:18:02.858 INFO:teuthology.orchestra.run.trial068.stderr:dumped all 2026-02-25T10:18:02.868 INFO:teuthology.orchestra.run.trial068.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-02-25T10:18:02.529220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4144,"kb_used_data":4144,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662480,"statfs":{"total":213674622976,"available":213670379520,"internally_reserved":0,"allocated":4243456,"data_stored":4243456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.971266"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567600+0000","last_change":"2026-02-25T10:17:59.570280+0000","last_active":"2026-02-25T10:17:59.567600+0000","last_peered":"2026-02-25T10:17:59.567600+0000","last_clean":"2026-02-25T10:17:59.567600+0000","last_became_active":"2026-02-25T10:17:57.603608+0000","last_became_peered":"2026-02-25T10:17:57.603608+0000","last_unstale":"2026-02-25T10:17:59.567600+0000","last_undegraded":"2026-02-25T10:17:59.567600+0000","last_fullsized":"2026-02-25T10:17:59.567600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567661+0000","last_change":"2026-02-25T10:17:59.569854+0000","last_active":"2026-02-25T10:17:59.567661+0000","last_peered":"2026-02-25T10:17:59.567661+0000","last_clean":"2026-02-25T10:17:59.567661+0000","last_became_active":"2026-02-25T10:17:57.594227+0000","last_became_peered":"2026-02-25T10:17:57.594227+0000","last_unstale":"2026-02-25T10:17:59.567661+0000","last_undegraded":"2026-02-25T10:17:59.567661+0000","last_fullsized":"2026-02-25T10:17:59.567661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567832+0000","last_change":"2026-02-25T10:17:59.570677+0000","last_active":"2026-02-25T10:17:59.567832+0000","last_peered":"2026-02-25T10:17:59.567832+0000","last_clean":"2026-02-25T10:17:59.567832+0000","last_became_active":"2026-02-25T10:17:57.592795+0000","last_became_peered":"2026-02-25T10:17:57.592795+0000","last_unstale":"2026-02-25T10:17:59.567832+0000","last_undegraded":"2026-02-25T10:17:59.567832+0000","last_fullsized":"2026-02-25T10:17:59.567832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.566066+0000","last_change":"2026-02-25T10:17:59.568986+0000","last_active":"2026-02-25T10:17:59.566066+0000","last_peered":"2026-02-25T10:17:59.566066+0000","last_clean":"2026-02-25T10:17:59.566066+0000","last_became_active":"2026-02-25T10:17:57.594095+0000","last_became_peered":"2026-02-25T10:17:57.594095+0000","last_unstale":"2026-02-25T10:17:59.566066+0000","last_undegraded":"2026-02-25T10:17:59.566066+0000","last_fullsized":"2026-02-25T10:17:59.566066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568519+0000","last_change":"2026-02-25T10:17:57.612424+0000","last_active":"2026-02-25T10:17:59.568519+0000","last_peered":"2026-02-25T10:17:59.568519+0000","last_clean":"2026-02-25T10:17:59.568519+0000","last_became_active":"2026-02-25T10:17:57.597495+0000","last_became_peered":"2026-02-25T10:17:57.597495+0000","last_unstale":"2026-02-25T10:17:59.568519+0000","last_undegraded":"2026-02-25T10:17:59.568519+0000","last_fullsized":"2026-02-25T10:17:59.568519+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.567310+0000","last_change":"2026-02-25T10:17:59.569701+0000","last_active":"2026-02-25T10:17:59.567310+0000","last_peered":"2026-02-25T10:17:59.567310+0000","last_clean":"2026-02-25T10:17:59.567310+0000","last_became_active":"2026-02-25T10:17:57.594174+0000","last_became_peered":"2026-02-25T10:17:57.594174+0000","last_unstale":"2026-02-25T10:17:59.567310+0000","last_undegraded":"2026-02-25T10:17:59.567310+0000","last_fullsized":"2026-02-25T10:17:59.567310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.568503+0000","last_change":"2026-02-25T10:17:59.568503+0000","last_active":"2026-02-25T10:17:59.568503+0000","last_peered":"2026-02-25T10:17:59.568503+0000","last_clean":"2026-02-25T10:17:59.568503+0000","last_became_active":"2026-02-25T10:17:57.607343+0000","last_became_peered":"2026-02-25T10:17:57.607343+0000","last_unstale":"2026-02-25T10:17:59.568503+0000","last_undegraded":"2026-02-25T10:17:59.568503+0000","last_fullsized":"2026-02-25T10:17:59.568503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.565888+0000","last_change":"2026-02-25T10:17:59.569065+0000","last_active":"2026-02-25T10:17:59.565888+0000","last_peered":"2026-02-25T10:17:59.565888+0000","last_clean":"2026-02-25T10:17:59.565888+0000","last_became_active":"2026-02-25T10:17:57.593096+0000","last_became_peered":"2026-02-25T10:17:57.593096+0000","last_unstale":"2026-02-25T10:17:59.565888+0000","last_undegraded":"2026-02-25T10:17:59.565888+0000","last_fullsized":"2026-02-25T10:17:59.565888+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T10:17:59.564834+0000","last_change":"2026-02-25T10:17:59.568550+0000","last_active":"2026-02-25T10:17:59.564834+0000","last_peered":"2026-02-25T10:17:59.564834+0000","last_clean":"2026-02-25T10:17:59.564834+0000","last_became_active":"2026-02-25T10:17:57.614238+0000","last_became_peered":"2026-02-25T10:17:57.614238+0000","last_unstale":"2026-02-25T10:17:59.564834+0000","last_undegraded":"2026-02-25T10:17:59.564834+0000","last_fullsized":"2026-02-25T10:17:59.564834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T10:17:56.546832+0000","last_clean_scrub_stamp":"2026-02-25T10:17:56.546832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670371328,"internally_reserved":0,"allocated":4251648,"data_stored":4251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":448,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166208,"statfs":{"total":53418655744,"available":53418196992,"internally_reserved":0,"allocated":458752,"data_stored":458752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1236,"kb_used_data":1236,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165420,"statfs":{"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1228,"kb_used_data":1228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165428,"statfs":{"total":53418655744,"available":53417398272,"internally_reserved":0,"allocated":1257472,"data_stored":1257472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1232,"kb_used_data":1232,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165424,"statfs":{"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53418655744,"available":53417390080,"internally_reserved":0,"allocated":1265664,"data_stored":1265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53418655744,"available":53418196992,"internally_reserved":0,"allocated":458752,"data_stored":458752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417394176,"internally_reserved":0,"allocated":1261568,"data_stored":1261568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T10:18:03.716 INFO:teuthology.orchestra.run.trial068.stderr:pool 'unique_pool_0' created 2026-02-25T10:18:03.730 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-25T10:18:03.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:03.806+0000 7f225d65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:03.816 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:03.814+0000 7f225d65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:04.719 INFO:teuthology.orchestra.run.trial068.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-25T10:18:04.805 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:18:04.886 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:04.883+0000 7f926ca8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:04.888 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:04.887+0000 7f926ca8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:05.736 DEBUG:teuthology.orchestra.run.trial068:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout:adding op weight read -> 100 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout:adding op weight write -> 50 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout:adding op weight delete -> 10 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout:adding op weight write_excl -> 50 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout:ceph version 20.3.0-5338-g6aeb6975 (6aeb6975bc9a4790a399c9cc70dba9635d2a7819) tentacle (dev - Debug) 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout:Configuration: 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout: Number of operations: 4000 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout: Number of objects: 500 2026-02-25T10:18:05.784 INFO:tasks.rados.rados.0.trial068.stdout: Max in flight operations: 16 2026-02-25T10:18:05.785 INFO:tasks.rados.rados.0.trial068.stdout: Object size (in bytes): 4000000 2026-02-25T10:18:05.785 INFO:tasks.rados.rados.0.trial068.stdout: Write stride min: 400000 2026-02-25T10:18:05.785 INFO:tasks.rados.rados.0.trial068.stdout: Write stride max: 800000 2026-02-25T10:18:06.725 INFO:tasks.rados.rados.0.trial068.stdout:1: write initial oid 1 2026-02-25T10:18:06.726 INFO:tasks.rados.rados.0.trial068.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2026-02-25T10:18:06.729 INFO:tasks.rados.rados.0.trial068.stdout:1: writing trial06842131-1 from 416807 to 892056 tid 1 2026-02-25T10:18:06.734 INFO:tasks.rados.rados.0.trial068.stdout:1: writing trial06842131-1 from 1542129 to 2016807 tid 2 2026-02-25T10:18:06.734 INFO:tasks.rados.rados.0.trial068.stdout:2: write initial oid 2 2026-02-25T10:18:06.734 INFO:tasks.rados.rados.0.trial068.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2026-02-25T10:18:06.737 INFO:tasks.rados.rados.0.trial068.stdout:2: writing trial06842131-2 from 416807 to 892056 tid 1 2026-02-25T10:18:06.742 INFO:tasks.rados.rados.0.trial068.stdout:2: writing trial06842131-2 from 1542129 to 2016807 tid 2 2026-02-25T10:18:06.742 INFO:tasks.rados.rados.0.trial068.stdout:3: write initial oid 3 2026-02-25T10:18:06.742 INFO:tasks.rados.rados.0.trial068.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2026-02-25T10:18:06.746 INFO:tasks.rados.rados.0.trial068.stdout:3: writing trial06842131-3 from 433614 to 984112 tid 1 2026-02-25T10:18:06.750 INFO:tasks.rados.rados.0.trial068.stdout:3: writing trial06842131-3 from 1600611 to 2033614 tid 2 2026-02-25T10:18:06.750 INFO:tasks.rados.rados.0.trial068.stdout:4: write initial oid 4 2026-02-25T10:18:06.750 INFO:tasks.rados.rados.0.trial068.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2026-02-25T10:18:06.754 INFO:tasks.rados.rados.0.trial068.stdout:4: writing trial06842131-4 from 450421 to 1076168 tid 1 2026-02-25T10:18:06.758 INFO:tasks.rados.rados.0.trial068.stdout:4: writing trial06842131-4 from 1659093 to 2050421 tid 2 2026-02-25T10:18:06.758 INFO:tasks.rados.rados.0.trial068.stdout:5: write initial oid 5 2026-02-25T10:18:06.758 INFO:tasks.rados.rados.0.trial068.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2026-02-25T10:18:06.762 INFO:tasks.rados.rados.0.trial068.stdout:5: writing trial06842131-5 from 467228 to 1168224 tid 1 2026-02-25T10:18:06.766 INFO:tasks.rados.rados.0.trial068.stdout:5: writing trial06842131-5 from 1717575 to 2067228 tid 2 2026-02-25T10:18:06.766 INFO:tasks.rados.rados.0.trial068.stdout:6: write initial oid 6 2026-02-25T10:18:06.766 INFO:tasks.rados.rados.0.trial068.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2026-02-25T10:18:06.771 INFO:tasks.rados.rados.0.trial068.stdout:6: writing trial06842131-6 from 484035 to 1260280 tid 1 2026-02-25T10:18:06.774 INFO:tasks.rados.rados.0.trial068.stdout:6: writing trial06842131-6 from 2059704 to 2084035 tid 2 2026-02-25T10:18:06.774 INFO:tasks.rados.rados.0.trial068.stdout:7: write initial oid 7 2026-02-25T10:18:06.774 INFO:tasks.rados.rados.0.trial068.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2026-02-25T10:18:06.778 INFO:tasks.rados.rados.0.trial068.stdout:7: writing trial06842131-7 from 500842 to 952336 tid 1 2026-02-25T10:18:06.783 INFO:tasks.rados.rados.0.trial068.stdout:7: writing trial06842131-7 from 1718186 to 2100842 tid 2 2026-02-25T10:18:06.783 INFO:tasks.rados.rados.0.trial068.stdout:8: write initial oid 8 2026-02-25T10:18:06.783 INFO:tasks.rados.rados.0.trial068.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2026-02-25T10:18:06.787 INFO:tasks.rados.rados.0.trial068.stdout:8: writing trial06842131-8 from 517649 to 1044392 tid 1 2026-02-25T10:18:06.791 INFO:tasks.rados.rados.0.trial068.stdout:8: writing trial06842131-8 from 1776668 to 2117649 tid 2 2026-02-25T10:18:06.791 INFO:tasks.rados.rados.0.trial068.stdout:9: write initial oid 9 2026-02-25T10:18:06.791 INFO:tasks.rados.rados.0.trial068.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2026-02-25T10:18:06.796 INFO:tasks.rados.rados.0.trial068.stdout:9: writing trial06842131-9 from 534456 to 1252801 tid 1 2026-02-25T10:18:06.800 INFO:tasks.rados.rados.0.trial068.stdout:9: writing trial06842131-9 from 1951503 to 2134456 tid 2 2026-02-25T10:18:06.800 INFO:tasks.rados.rados.0.trial068.stdout:10: write initial oid 10 2026-02-25T10:18:06.800 INFO:tasks.rados.rados.0.trial068.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2026-02-25T10:18:06.805 INFO:tasks.rados.rados.0.trial068.stdout:10: writing trial06842131-10 from 551263 to 1344857 tid 1 2026-02-25T10:18:06.808 INFO:tasks.rados.rados.0.trial068.stdout:10: writing trial06842131-10 from 1893632 to 2151263 tid 2 2026-02-25T10:18:06.808 INFO:tasks.rados.rados.0.trial068.stdout:11: write initial oid 11 2026-02-25T10:18:06.808 INFO:tasks.rados.rados.0.trial068.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2026-02-25T10:18:06.812 INFO:tasks.rados.rados.0.trial068.stdout:11: writing trial06842131-11 from 568070 to 1036913 tid 1 2026-02-25T10:18:06.816 INFO:tasks.rados.rados.0.trial068.stdout:11: writing trial06842131-11 from 1552114 to 2168070 tid 2 2026-02-25T10:18:06.816 INFO:tasks.rados.rados.0.trial068.stdout:12: write initial oid 12 2026-02-25T10:18:06.816 INFO:tasks.rados.rados.0.trial068.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2026-02-25T10:18:06.820 INFO:tasks.rados.rados.0.trial068.stdout:12: writing trial06842131-12 from 584877 to 1128969 tid 1 2026-02-25T10:18:06.824 INFO:tasks.rados.rados.0.trial068.stdout:12: writing trial06842131-12 from 1610596 to 2172599 tid 2 2026-02-25T10:18:06.825 INFO:tasks.rados.rados.0.trial068.stdout:13: write initial oid 13 2026-02-25T10:18:06.825 INFO:tasks.rados.rados.0.trial068.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2026-02-25T10:18:06.829 INFO:tasks.rados.rados.0.trial068.stdout:13: writing trial06842131-13 from 601684 to 1221025 tid 1 2026-02-25T10:18:06.833 INFO:tasks.rados.rados.0.trial068.stdout:13: writing trial06842131-13 from 1669078 to 2201684 tid 2 2026-02-25T10:18:06.833 INFO:tasks.rados.rados.0.trial068.stdout:14: write initial oid 14 2026-02-25T10:18:06.833 INFO:tasks.rados.rados.0.trial068.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2026-02-25T10:18:06.838 INFO:tasks.rados.rados.0.trial068.stdout:14: writing trial06842131-14 from 618491 to 1313081 tid 1 2026-02-25T10:18:06.842 INFO:tasks.rados.rados.0.trial068.stdout:14: writing trial06842131-14 from 2011207 to 2218491 tid 2 2026-02-25T10:18:06.842 INFO:tasks.rados.rados.0.trial068.stdout:15: write initial oid 15 2026-02-25T10:18:06.842 INFO:tasks.rados.rados.0.trial068.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2026-02-25T10:18:06.847 INFO:tasks.rados.rados.0.trial068.stdout:15: writing trial06842131-15 from 635298 to 1405137 tid 1 2026-02-25T10:18:06.850 INFO:tasks.rados.rados.0.trial068.stdout:15: writing trial06842131-15 from 2069689 to 2235298 tid 2 2026-02-25T10:18:06.850 INFO:tasks.rados.rados.0.trial068.stdout:16: write initial oid 16 2026-02-25T10:18:06.850 INFO:tasks.rados.rados.0.trial068.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2026-02-25T10:18:06.855 INFO:tasks.rados.rados.0.trial068.stdout:16: writing trial06842131-16 from 652105 to 1097193 tid 1 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:16: writing trial06842131-16 from 1728171 to 2181159 tid 2 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:1: finishing write tid 1 to trial06842131-1 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:4: finishing write tid 1 to trial06842131-4 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: finishing write tid 1 to trial06842131-5 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:3: finishing write tid 1 to trial06842131-3 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:6: finishing write tid 1 to trial06842131-6 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: finishing write tid 2 to trial06842131-5 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: finishing write tid 3 to trial06842131-5 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: finishing write tid 4 to trial06842131-5 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: oid 5 updating version 0 to 1 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: oid 5 updating version 1 to 3 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: oid 5 version 3 is already newer than 3 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:1: finishing write tid 2 to trial06842131-1 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:4: finishing write tid 2 to trial06842131-4 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:1: finishing write tid 3 to trial06842131-1 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:6: finishing write tid 2 to trial06842131-6 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:4: finishing write tid 3 to trial06842131-4 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:5: done (15 left) 2026-02-25T10:18:06.859 INFO:tasks.rados.rados.0.trial068.stdout:17: write initial oid 17 2026-02-25T10:18:06.860 INFO:tasks.rados.rados.0.trial068.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2026-02-25T10:18:06.864 INFO:tasks.rados.rados.0.trial068.stdout:17: writing trial06842131-17 from 668912 to 1305602 tid 1 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:17: writing trial06842131-17 from 1903006 to 2268912 tid 2 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:6: finishing write tid 3 to trial06842131-6 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:1: finishing write tid 4 to trial06842131-1 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:1: oid 1 updating version 0 to 1 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:1: oid 1 updating version 1 to 3 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:1: oid 1 version 3 is already newer than 3 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:6: finishing write tid 4 to trial06842131-6 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:6: oid 6 updating version 0 to 1 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:6: oid 6 updating version 1 to 3 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:6: oid 6 version 3 is already newer than 3 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:4: finishing write tid 4 to trial06842131-4 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:4: oid 4 updating version 0 to 3 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:4: oid 4 version 3 is already newer than 1 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:4: oid 4 version 3 is already newer than 3 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2026-02-25T10:18:06.868 INFO:tasks.rados.rados.0.trial068.stdout:3: finishing write tid 2 to trial06842131-3 2026-02-25T10:18:06.869 INFO:tasks.rados.rados.0.trial068.stdout:11: finishing write tid 1 to trial06842131-11 2026-02-25T10:18:06.869 INFO:tasks.rados.rados.0.trial068.stdout:1: done (15 left) 2026-02-25T10:18:06.869 INFO:tasks.rados.rados.0.trial068.stdout:4: done (14 left) 2026-02-25T10:18:06.869 INFO:tasks.rados.rados.0.trial068.stdout:6: done (13 left) 2026-02-25T10:18:06.869 INFO:tasks.rados.rados.0.trial068.stdout:18: write initial oid 18 2026-02-25T10:18:06.869 INFO:tasks.rados.rados.0.trial068.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2026-02-25T10:18:06.873 INFO:tasks.rados.rados.0.trial068.stdout:18: writing trial06842131-18 from 685719 to 1397658 tid 1 2026-02-25T10:18:06.877 INFO:tasks.rados.rados.0.trial068.stdout:18: writing trial06842131-18 from 1845135 to 2285719 tid 2 2026-02-25T10:18:06.877 INFO:tasks.rados.rados.0.trial068.stdout:19: write initial oid 19 2026-02-25T10:18:06.877 INFO:tasks.rados.rados.0.trial068.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2026-02-25T10:18:06.882 INFO:tasks.rados.rados.0.trial068.stdout:19: writing trial06842131-19 from 702526 to 1489714 tid 1 2026-02-25T10:18:06.885 INFO:tasks.rados.rados.0.trial068.stdout:19: writing trial06842131-19 from 1903617 to 2302526 tid 2 2026-02-25T10:18:06.885 INFO:tasks.rados.rados.0.trial068.stdout:20: write initial oid 20 2026-02-25T10:18:06.885 INFO:tasks.rados.rados.0.trial068.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2026-02-25T10:18:06.890 INFO:tasks.rados.rados.0.trial068.stdout:20: writing trial06842131-20 from 719333 to 1181770 tid 1 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:20: writing trial06842131-20 from 1962099 to 2319333 tid 2 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:8: finishing write tid 1 to trial06842131-8 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:9: finishing write tid 1 to trial06842131-9 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:3: finishing write tid 3 to trial06842131-3 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:3: finishing write tid 4 to trial06842131-3 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:3: oid 3 updating version 0 to 1 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:3: oid 3 updating version 1 to 3 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:3: oid 3 version 3 is already newer than 3 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2026-02-25T10:18:06.894 INFO:tasks.rados.rados.0.trial068.stdout:12: finishing write tid 1 to trial06842131-12 2026-02-25T10:18:06.895 INFO:tasks.rados.rados.0.trial068.stdout:9: finishing write tid 2 to trial06842131-9 2026-02-25T10:18:06.895 INFO:tasks.rados.rados.0.trial068.stdout:10: finishing write tid 1 to trial06842131-10 2026-02-25T10:18:06.895 INFO:tasks.rados.rados.0.trial068.stdout:3: done (15 left) 2026-02-25T10:18:06.895 INFO:tasks.rados.rados.0.trial068.stdout:21: write initial oid 21 2026-02-25T10:18:06.895 INFO:tasks.rados.rados.0.trial068.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2026-02-25T10:18:06.899 INFO:tasks.rados.rados.0.trial068.stdout:21: writing trial06842131-21 from 736140 to 1273826 tid 1 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:21: writing trial06842131-21 from 2020581 to 2336140 tid 2 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:9: finishing write tid 3 to trial06842131-9 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:2: finishing write tid 1 to trial06842131-2 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:11: finishing write tid 2 to trial06842131-11 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:8: finishing write tid 2 to trial06842131-8 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:9: finishing write tid 4 to trial06842131-9 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:9: oid 9 updating version 0 to 3 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:9: oid 9 version 3 is already newer than 3 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:9: oid 9 version 3 is already newer than 1 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:9: done (15 left) 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:22: write initial oid 22 2026-02-25T10:18:06.903 INFO:tasks.rados.rados.0.trial068.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2026-02-25T10:18:06.908 INFO:tasks.rados.rados.0.trial068.stdout:22: writing trial06842131-22 from 752947 to 1365882 tid 1 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:22: writing trial06842131-22 from 1962710 to 2352947 tid 2 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:7: finishing write tid 1 to trial06842131-7 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:12: finishing write tid 2 to trial06842131-12 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:8: finishing write tid 3 to trial06842131-8 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:11: finishing write tid 3 to trial06842131-11 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:8: finishing write tid 4 to trial06842131-8 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:8: oid 8 updating version 0 to 3 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:8: oid 8 version 3 is already newer than 3 2026-02-25T10:18:06.912 INFO:tasks.rados.rados.0.trial068.stdout:8: oid 8 version 3 is already newer than 1 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:11: finishing write tid 4 to trial06842131-11 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:11: oid 11 updating version 0 to 3 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:11: oid 11 version 3 is already newer than 3 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:11: oid 11 version 3 is already newer than 1 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:12: finishing write tid 3 to trial06842131-12 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:12: finishing write tid 4 to trial06842131-12 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:12: oid 12 updating version 0 to 1 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:12: oid 12 updating version 1 to 3 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:12: oid 12 version 3 is already newer than 3 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:14: finishing write tid 1 to trial06842131-14 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:10: finishing write tid 2 to trial06842131-10 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:13: finishing write tid 1 to trial06842131-13 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:8: done (15 left) 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:11: done (14 left) 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:12: done (13 left) 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:23: write initial oid 23 2026-02-25T10:18:06.913 INFO:tasks.rados.rados.0.trial068.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2026-02-25T10:18:06.918 INFO:tasks.rados.rados.0.trial068.stdout:23: writing trial06842131-23 from 769754 to 1457938 tid 1 2026-02-25T10:18:06.921 INFO:tasks.rados.rados.0.trial068.stdout:23: writing trial06842131-23 from 2021192 to 2369754 tid 2 2026-02-25T10:18:06.921 INFO:tasks.rados.rados.0.trial068.stdout:24: write initial oid 24 2026-02-25T10:18:06.921 INFO:tasks.rados.rados.0.trial068.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2026-02-25T10:18:06.926 INFO:tasks.rados.rados.0.trial068.stdout:24: writing trial06842131-24 from 786561 to 1266347 tid 1 2026-02-25T10:18:06.930 INFO:tasks.rados.rados.0.trial068.stdout:24: writing trial06842131-24 from 1796027 to 2263691 tid 2 2026-02-25T10:18:06.930 INFO:tasks.rados.rados.0.trial068.stdout:25: write initial oid 25 2026-02-25T10:18:06.930 INFO:tasks.rados.rados.0.trial068.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2026-02-25T10:18:06.934 INFO:tasks.rados.rados.0.trial068.stdout:25: writing trial06842131-25 from 403368 to 958403 tid 1 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:25: writing trial06842131-25 from 1454509 to 2182184 tid 2 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:18: finishing write tid 1 to trial06842131-18 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: finishing write tid 2 to trial06842131-14 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:19: finishing write tid 1 to trial06842131-19 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:7: finishing write tid 2 to trial06842131-7 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: finishing write tid 3 to trial06842131-14 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: finishing write tid 4 to trial06842131-14 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: oid 14 updating version 0 to 6 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: oid 14 version 6 is already newer than 4 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: oid 14 version 6 is already newer than 6 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:13: finishing write tid 2 to trial06842131-13 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:7: finishing write tid 3 to trial06842131-7 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:14: done (15 left) 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:26: write initial oid 26 2026-02-25T10:18:06.939 INFO:tasks.rados.rados.0.trial068.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2026-02-25T10:18:06.943 INFO:tasks.rados.rados.0.trial068.stdout:26: writing trial06842131-26 from 420175 to 1050459 tid 1 2026-02-25T10:18:06.947 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:18:06.948 INFO:tasks.rados.rados.0.trial068.stdout:26: writing trial06842131-26 from 1796638 to 2267971 tid 2 2026-02-25T10:18:06.948 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.948 INFO:tasks.rados.rados.0.trial068.stdout:13: finishing write tid 3 to trial06842131-13 2026-02-25T10:18:06.948 INFO:tasks.rados.rados.0.trial068.stdout:7: finishing write tid 4 to trial06842131-7 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:7: oid 7 updating version 0 to 3 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:7: oid 7 version 3 is already newer than 3 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:7: oid 7 version 3 is already newer than 1 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:18: finishing write tid 2 to trial06842131-18 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:7: done (15 left) 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:27: write initial oid 27 2026-02-25T10:18:06.949 INFO:tasks.rados.rados.0.trial068.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2026-02-25T10:18:06.952 INFO:tasks.rados.rados.0.trial068.stdout:27: writing trial06842131-27 from 436982 to 1142515 tid 1 2026-02-25T10:18:06.957 INFO:tasks.rados.rados.0.trial068.stdout:27: writing trial06842131-27 from 1855120 to 2436982 tid 2 2026-02-25T10:18:06.957 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:13: finishing write tid 4 to trial06842131-13 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:13: oid 13 updating version 0 to 3 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:13: oid 13 version 3 is already newer than 1 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:13: oid 13 version 3 is already newer than 3 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:13: done (15 left) 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:28: write initial oid 28 2026-02-25T10:18:06.958 INFO:tasks.rados.rados.0.trial068.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2026-02-25T10:18:06.962 INFO:tasks.rados.rados.0.trial068.stdout:28: writing trial06842131-28 from 453789 to 1234571 tid 1 2026-02-25T10:18:06.967 INFO:tasks.rados.rados.0.trial068.stdout:28: writing trial06842131-28 from 1913602 to 2388604 tid 2 2026-02-25T10:18:06.967 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.967 INFO:tasks.rados.rados.0.trial068.stdout:22: finishing write tid 1 to trial06842131-22 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:16: finishing write tid 1 to trial06842131-16 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:18: finishing write tid 3 to trial06842131-18 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:18: finishing write tid 4 to trial06842131-18 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:18: oid 18 updating version 0 to 6 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:18: oid 18 version 6 is already newer than 4 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:18: oid 18 version 6 is already newer than 6 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:17: finishing write tid 1 to trial06842131-17 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:18: done (15 left) 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:29: write initial oid 29 2026-02-25T10:18:06.968 INFO:tasks.rados.rados.0.trial068.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2026-02-25T10:18:06.970 INFO:tasks.rados.rados.0.trial068.stdout:29: writing trial06842131-29 from 470596 to 926627 tid 1 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:29: writing trial06842131-29 from 1572084 to 2190744 tid 2 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:23: finishing write tid 1 to trial06842131-23 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:20: finishing write tid 1 to trial06842131-20 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:19: finishing write tid 2 to trial06842131-19 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:22: finishing write tid 2 to trial06842131-22 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:19: finishing write tid 3 to trial06842131-19 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:24: finishing write tid 1 to trial06842131-24 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:19: finishing write tid 4 to trial06842131-19 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:19: oid 19 updating version 0 to 6 2026-02-25T10:18:06.977 INFO:tasks.rados.rados.0.trial068.stdout:19: oid 19 version 6 is already newer than 6 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:19: oid 19 version 6 is already newer than 4 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:10: finishing write tid 3 to trial06842131-10 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:10: finishing write tid 4 to trial06842131-10 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:10: oid 10 updating version 0 to 4 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:10: oid 10 updating version 4 to 6 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:10: oid 10 version 6 is already newer than 6 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:22: finishing write tid 3 to trial06842131-22 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:10: done (15 left) 2026-02-25T10:18:06.978 INFO:tasks.rados.rados.0.trial068.stdout:19: done (14 left) 2026-02-25T10:18:06.979 INFO:tasks.rados.rados.0.trial068.stdout:30: write initial oid 30 2026-02-25T10:18:06.979 INFO:tasks.rados.rados.0.trial068.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2026-02-25T10:18:06.980 INFO:tasks.rados.rados.0.trial068.stdout:30: writing trial06842131-30 from 487403 to 1018683 tid 1 2026-02-25T10:18:06.984 INFO:tasks.rados.rados.0.trial068.stdout:30: writing trial06842131-30 from 1514213 to 1992884 tid 2 2026-02-25T10:18:06.984 INFO:tasks.rados.rados.0.trial068.stdout:31: write initial oid 31 2026-02-25T10:18:06.984 INFO:tasks.rados.rados.0.trial068.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2026-02-25T10:18:06.989 INFO:tasks.rados.rados.0.trial068.stdout:31: writing trial06842131-31 from 504210 to 1110739 tid 1 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:31: writing trial06842131-31 from 1572695 to 2195024 tid 2 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:23: finishing write tid 2 to trial06842131-23 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:22: finishing write tid 4 to trial06842131-22 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:22: oid 22 updating version 0 to 1 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:22: oid 22 updating version 1 to 3 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:22: oid 22 version 3 is already newer than 3 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:26: finishing write tid 1 to trial06842131-26 2026-02-25T10:18:06.995 INFO:tasks.rados.rados.0.trial068.stdout:22: done (15 left) 2026-02-25T10:18:06.996 INFO:tasks.rados.rados.0.trial068.stdout:32: write initial oid 32 2026-02-25T10:18:06.996 INFO:tasks.rados.rados.0.trial068.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2026-02-25T10:18:06.999 INFO:tasks.rados.rados.0.trial068.stdout:32: writing trial06842131-32 from 521017 to 1319148 tid 1 2026-02-25T10:18:07.004 INFO:tasks.rados.rados.0.trial068.stdout:32: writing trial06842131-32 from 1747530 to 2229870 tid 2 2026-02-25T10:18:07.004 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.004 INFO:tasks.rados.rados.0.trial068.stdout:2: finishing write tid 2 to trial06842131-2 2026-02-25T10:18:07.004 INFO:tasks.rados.rados.0.trial068.stdout:27: finishing write tid 1 to trial06842131-27 2026-02-25T10:18:07.004 INFO:tasks.rados.rados.0.trial068.stdout:20: finishing write tid 2 to trial06842131-20 2026-02-25T10:18:07.004 INFO:tasks.rados.rados.0.trial068.stdout:2: finishing write tid 3 to trial06842131-2 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:23: finishing write tid 3 to trial06842131-23 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:2: finishing write tid 4 to trial06842131-2 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:2: oid 2 updating version 0 to 3 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:2: oid 2 version 3 is already newer than 1 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:2: oid 2 version 3 is already newer than 3 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:15: finishing write tid 1 to trial06842131-15 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:20: finishing write tid 3 to trial06842131-20 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:23: finishing write tid 4 to trial06842131-23 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:23: oid 23 updating version 0 to 9 2026-02-25T10:18:07.005 INFO:tasks.rados.rados.0.trial068.stdout:23: oid 23 version 9 is already newer than 7 2026-02-25T10:18:07.006 INFO:tasks.rados.rados.0.trial068.stdout:23: oid 23 version 9 is already newer than 9 2026-02-25T10:18:07.006 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2026-02-25T10:18:07.006 INFO:tasks.rados.rados.0.trial068.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2026-02-25T10:18:07.006 INFO:tasks.rados.rados.0.trial068.stdout:2: done (15 left) 2026-02-25T10:18:07.006 INFO:tasks.rados.rados.0.trial068.stdout:23: done (14 left) 2026-02-25T10:18:07.006 INFO:tasks.rados.rados.0.trial068.stdout:33: write initial oid 33 2026-02-25T10:18:07.006 INFO:tasks.rados.rados.0.trial068.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2026-02-25T10:18:07.007 INFO:tasks.rados.rados.0.trial068.stdout:33: writing trial06842131-33 from 537824 to 1011204 tid 1 2026-02-25T10:18:07.013 INFO:tasks.rados.rados.0.trial068.stdout:33: writing trial06842131-33 from 1806012 to 2432010 tid 2 2026-02-25T10:18:07.013 INFO:tasks.rados.rados.0.trial068.stdout:34: write initial oid 34 2026-02-25T10:18:07.014 INFO:tasks.rados.rados.0.trial068.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2026-02-25T10:18:07.018 INFO:tasks.rados.rados.0.trial068.stdout:34: writing trial06842131-34 from 554631 to 1103260 tid 1 2026-02-25T10:18:07.023 INFO:tasks.rados.rados.0.trial068.stdout:34: writing trial06842131-34 from 1748141 to 2234150 tid 2 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:20: finishing write tid 4 to trial06842131-20 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:20: oid 20 updating version 0 to 9 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:20: oid 20 version 9 is already newer than 9 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:20: oid 20 version 9 is already newer than 7 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:20: done (15 left) 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:35: write initial oid 35 2026-02-25T10:18:07.024 INFO:tasks.rados.rados.0.trial068.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2026-02-25T10:18:07.027 INFO:tasks.rados.rados.0.trial068.stdout:35: writing trial06842131-35 from 571438 to 1195316 tid 1 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout:35: writing trial06842131-35 from 1806623 to 2436290 tid 2 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout:17: finishing write tid 2 to trial06842131-17 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout:29: finishing write tid 1 to trial06842131-29 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout:30: finishing write tid 1 to trial06842131-30 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout:27: finishing write tid 2 to trial06842131-27 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout:27: finishing write tid 3 to trial06842131-27 2026-02-25T10:18:07.033 INFO:tasks.rados.rados.0.trial068.stdout:26: finishing write tid 2 to trial06842131-26 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:27: finishing write tid 4 to trial06842131-27 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:27: oid 27 updating version 0 to 6 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:27: oid 27 version 6 is already newer than 6 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:27: oid 27 version 6 is already newer than 4 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:24: finishing write tid 2 to trial06842131-24 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:27: done (15 left) 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:36: write initial oid 36 2026-02-25T10:18:07.034 INFO:tasks.rados.rados.0.trial068.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2026-02-25T10:18:07.035 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:07.032+0000 7f34d6393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:07.037 INFO:tasks.rados.rados.0.trial068.stdout:36: writing trial06842131-36 from 588245 to 1287372 tid 1 2026-02-25T10:18:07.038 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:07.037+0000 7f34d6393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:07.042 INFO:tasks.rados.rados.0.trial068.stdout:36: writing trial06842131-36 from 1865105 to 2354783 tid 2 2026-02-25T10:18:07.042 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:17: finishing write tid 3 to trial06842131-17 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:26: finishing write tid 3 to trial06842131-26 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:30: finishing write tid 2 to trial06842131-30 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:17: finishing write tid 4 to trial06842131-17 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:17: oid 17 updating version 0 to 4 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:17: oid 17 updating version 4 to 6 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:17: oid 17 version 6 is already newer than 6 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:26: finishing write tid 4 to trial06842131-26 2026-02-25T10:18:07.043 INFO:tasks.rados.rados.0.trial068.stdout:26: oid 26 updating version 0 to 6 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:26: oid 26 version 6 is already newer than 6 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:26: oid 26 version 6 is already newer than 4 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:30: finishing write tid 3 to trial06842131-30 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:17: done (15 left) 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:26: done (14 left) 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:37: write initial oid 37 2026-02-25T10:18:07.044 INFO:tasks.rados.rados.0.trial068.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2026-02-25T10:18:07.047 INFO:tasks.rados.rados.0.trial068.stdout:37: writing trial06842131-37 from 605052 to 1379428 tid 1 2026-02-25T10:18:07.052 INFO:tasks.rados.rados.0.trial068.stdout:37: writing trial06842131-37 from 1923587 to 2556923 tid 2 2026-02-25T10:18:07.052 INFO:tasks.rados.rados.0.trial068.stdout:38: write initial oid 38 2026-02-25T10:18:07.052 INFO:tasks.rados.rados.0.trial068.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2026-02-25T10:18:07.056 INFO:tasks.rados.rados.0.trial068.stdout:38: writing trial06842131-38 from 621859 to 1071484 tid 1 2026-02-25T10:18:07.063 INFO:tasks.rados.rados.0.trial068.stdout:38: writing trial06842131-38 from 1865716 to 2621859 tid 2 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:34: finishing write tid 1 to trial06842131-34 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:30: finishing write tid 4 to trial06842131-30 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:30: oid 30 updating version 0 to 3 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:30: oid 30 version 3 is already newer than 3 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:30: oid 30 version 3 is already newer than 1 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:31: finishing write tid 1 to trial06842131-31 2026-02-25T10:18:07.064 INFO:tasks.rados.rados.0.trial068.stdout:30: done (15 left) 2026-02-25T10:18:07.065 INFO:tasks.rados.rados.0.trial068.stdout:39: write initial oid 39 2026-02-25T10:18:07.065 INFO:tasks.rados.rados.0.trial068.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2026-02-25T10:18:07.067 INFO:tasks.rados.rados.0.trial068.stdout:39: writing trial06842131-39 from 638666 to 1163540 tid 1 2026-02-25T10:18:07.073 INFO:tasks.rados.rados.0.trial068.stdout:39: writing trial06842131-39 from 1924198 to 2561203 tid 2 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:15: finishing write tid 2 to trial06842131-15 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:29: finishing write tid 2 to trial06842131-29 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:15: finishing write tid 3 to trial06842131-15 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:34: finishing write tid 2 to trial06842131-34 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:37: finishing write tid 1 to trial06842131-37 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:15: finishing write tid 4 to trial06842131-15 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:15: oid 15 updating version 0 to 6 2026-02-25T10:18:07.074 INFO:tasks.rados.rados.0.trial068.stdout:15: oid 15 version 6 is already newer than 6 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:15: oid 15 version 6 is already newer than 4 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:29: finishing write tid 3 to trial06842131-29 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:34: finishing write tid 3 to trial06842131-34 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:15: done (15 left) 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:40: write initial oid 40 2026-02-25T10:18:07.075 INFO:tasks.rados.rados.0.trial068.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2026-02-25T10:18:07.078 INFO:tasks.rados.rados.0.trial068.stdout:40: writing trial06842131-40 from 655473 to 1371949 tid 1 2026-02-25T10:18:07.084 INFO:tasks.rados.rados.0.trial068.stdout:40: writing trial06842131-40 from 2099033 to 2655473 tid 2 2026-02-25T10:18:07.084 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.084 INFO:tasks.rados.rados.0.trial068.stdout:34: finishing write tid 4 to trial06842131-34 2026-02-25T10:18:07.084 INFO:tasks.rados.rados.0.trial068.stdout:34: oid 34 updating version 0 to 3 2026-02-25T10:18:07.084 INFO:tasks.rados.rados.0.trial068.stdout:34: oid 34 version 3 is already newer than 1 2026-02-25T10:18:07.084 INFO:tasks.rados.rados.0.trial068.stdout:34: oid 34 version 3 is already newer than 3 2026-02-25T10:18:07.084 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2026-02-25T10:18:07.085 INFO:tasks.rados.rados.0.trial068.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2026-02-25T10:18:07.085 INFO:tasks.rados.rados.0.trial068.stdout:34: done (15 left) 2026-02-25T10:18:07.085 INFO:tasks.rados.rados.0.trial068.stdout:41: write initial oid 41 2026-02-25T10:18:07.085 INFO:tasks.rados.rados.0.trial068.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2026-02-25T10:18:07.089 INFO:tasks.rados.rados.0.trial068.stdout:41: writing trial06842131-41 from 672280 to 1464005 tid 1 2026-02-25T10:18:07.094 INFO:tasks.rados.rados.0.trial068.stdout:41: writing trial06842131-41 from 2157515 to 2672280 tid 2 2026-02-25T10:18:07.094 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.094 INFO:tasks.rados.rados.0.trial068.stdout:29: finishing write tid 4 to trial06842131-29 2026-02-25T10:18:07.094 INFO:tasks.rados.rados.0.trial068.stdout:29: oid 29 updating version 0 to 3 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:29: oid 29 version 3 is already newer than 1 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:29: oid 29 version 3 is already newer than 3 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:16: finishing write tid 2 to trial06842131-16 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:38: finishing write tid 1 to trial06842131-38 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:29: done (15 left) 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:42: write initial oid 42 2026-02-25T10:18:07.095 INFO:tasks.rados.rados.0.trial068.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2026-02-25T10:18:07.098 INFO:tasks.rados.rados.0.trial068.stdout:42: writing trial06842131-42 from 689087 to 1156061 tid 1 2026-02-25T10:18:07.104 INFO:tasks.rados.rados.0.trial068.stdout:42: writing trial06842131-42 from 1699644 to 2483976 tid 2 2026-02-25T10:18:07.104 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.104 INFO:tasks.rados.rados.0.trial068.stdout:35: finishing write tid 1 to trial06842131-35 2026-02-25T10:18:07.104 INFO:tasks.rados.rados.0.trial068.stdout:16: finishing write tid 3 to trial06842131-16 2026-02-25T10:18:07.104 INFO:tasks.rados.rados.0.trial068.stdout:16: finishing write tid 4 to trial06842131-16 2026-02-25T10:18:07.104 INFO:tasks.rados.rados.0.trial068.stdout:16: oid 16 updating version 0 to 6 2026-02-25T10:18:07.105 INFO:tasks.rados.rados.0.trial068.stdout:16: oid 16 version 6 is already newer than 6 2026-02-25T10:18:07.105 INFO:tasks.rados.rados.0.trial068.stdout:16: oid 16 version 6 is already newer than 4 2026-02-25T10:18:07.105 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2026-02-25T10:18:07.105 INFO:tasks.rados.rados.0.trial068.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2026-02-25T10:18:07.105 INFO:tasks.rados.rados.0.trial068.stdout:16: done (15 left) 2026-02-25T10:18:07.105 INFO:tasks.rados.rados.0.trial068.stdout:43: write initial oid 43 2026-02-25T10:18:07.105 INFO:tasks.rados.rados.0.trial068.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2026-02-25T10:18:07.108 INFO:tasks.rados.rados.0.trial068.stdout:43: writing trial06842131-43 from 705894 to 1248117 tid 1 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:43: writing trial06842131-43 from 1758126 to 2402469 tid 2 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:21: finishing write tid 1 to trial06842131-21 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:24: finishing write tid 3 to trial06842131-24 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:31: finishing write tid 2 to trial06842131-31 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:24: finishing write tid 4 to trial06842131-24 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:24: oid 24 updating version 0 to 6 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:24: oid 24 version 6 is already newer than 6 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:24: oid 24 version 6 is already newer than 4 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2026-02-25T10:18:07.113 INFO:tasks.rados.rados.0.trial068.stdout:38: finishing write tid 2 to trial06842131-38 2026-02-25T10:18:07.114 INFO:tasks.rados.rados.0.trial068.stdout:24: done (15 left) 2026-02-25T10:18:07.114 INFO:tasks.rados.rados.0.trial068.stdout:44: write initial oid 44 2026-02-25T10:18:07.114 INFO:tasks.rados.rados.0.trial068.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2026-02-25T10:18:07.118 INFO:tasks.rados.rados.0.trial068.stdout:44: writing trial06842131-44 from 722701 to 1340173 tid 1 2026-02-25T10:18:07.123 INFO:tasks.rados.rados.0.trial068.stdout:44: writing trial06842131-44 from 1816608 to 2604609 tid 2 2026-02-25T10:18:07.123 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.127 INFO:tasks.rados.rados.0.trial068.stdout:37: finishing write tid 2 to trial06842131-37 2026-02-25T10:18:07.129 INFO:tasks.rados.rados.0.trial068.stdout:38: finishing write tid 3 to trial06842131-38 2026-02-25T10:18:07.130 INFO:tasks.rados.rados.0.trial068.stdout:35: finishing write tid 2 to trial06842131-35 2026-02-25T10:18:07.132 INFO:tasks.rados.rados.0.trial068.stdout:28: finishing write tid 1 to trial06842131-28 2026-02-25T10:18:07.132 INFO:tasks.rados.rados.0.trial068.stdout:38: finishing write tid 4 to trial06842131-38 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:38: oid 38 updating version 0 to 4 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:38: oid 38 updating version 4 to 6 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:38: oid 38 version 6 is already newer than 6 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:38: done (15 left) 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:45: write initial oid 45 2026-02-25T10:18:07.133 INFO:tasks.rados.rados.0.trial068.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2026-02-25T10:18:07.138 INFO:tasks.rados.rados.0.trial068.stdout:45: writing trial06842131-45 from 739508 to 1432229 tid 1 2026-02-25T10:18:07.143 INFO:tasks.rados.rados.0.trial068.stdout:45: writing trial06842131-45 from 1875090 to 2523102 tid 2 2026-02-25T10:18:07.143 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.143 INFO:tasks.rados.rados.0.trial068.stdout:21: finishing write tid 2 to trial06842131-21 2026-02-25T10:18:07.144 INFO:tasks.rados.rados.0.trial068.stdout:39: finishing write tid 1 to trial06842131-39 2026-02-25T10:18:07.147 INFO:tasks.rados.rados.0.trial068.stdout:41: finishing write tid 1 to trial06842131-41 2026-02-25T10:18:07.157 INFO:tasks.rados.rados.0.trial068.stdout:35: finishing write tid 3 to trial06842131-35 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:37: finishing write tid 3 to trial06842131-37 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:35: finishing write tid 4 to trial06842131-35 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:35: oid 35 updating version 0 to 4 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:35: oid 35 updating version 4 to 6 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:35: oid 35 version 6 is already newer than 6 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:35: done (15 left) 2026-02-25T10:18:07.162 INFO:tasks.rados.rados.0.trial068.stdout:46: write initial oid 46 2026-02-25T10:18:07.163 INFO:tasks.rados.rados.0.trial068.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2026-02-25T10:18:07.168 INFO:tasks.rados.rados.0.trial068.stdout:46: writing trial06842131-46 from 756315 to 1524285 tid 1 2026-02-25T10:18:07.173 INFO:tasks.rados.rados.0.trial068.stdout:46: writing trial06842131-46 from 1933572 to 2725242 tid 2 2026-02-25T10:18:07.173 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.173 INFO:tasks.rados.rados.0.trial068.stdout:41: finishing write tid 2 to trial06842131-41 2026-02-25T10:18:07.173 INFO:tasks.rados.rados.0.trial068.stdout:21: finishing write tid 3 to trial06842131-21 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:37: finishing write tid 4 to trial06842131-37 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:37: oid 37 updating version 0 to 6 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:37: oid 37 version 6 is already newer than 4 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:37: oid 37 version 6 is already newer than 6 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:21: finishing write tid 4 to trial06842131-21 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:21: oid 21 updating version 0 to 9 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:21: oid 21 version 9 is already newer than 7 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:21: oid 21 version 9 is already newer than 9 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:21: done (15 left) 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:37: done (14 left) 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:47: write initial oid 47 2026-02-25T10:18:07.174 INFO:tasks.rados.rados.0.trial068.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2026-02-25T10:18:07.179 INFO:tasks.rados.rados.0.trial068.stdout:47: writing trial06842131-47 from 773122 to 1332694 tid 1 2026-02-25T10:18:07.184 INFO:tasks.rados.rados.0.trial068.stdout:47: writing trial06842131-47 from 1992054 to 2643735 tid 2 2026-02-25T10:18:07.184 INFO:tasks.rados.rados.0.trial068.stdout:48: write initial oid 48 2026-02-25T10:18:07.184 INFO:tasks.rados.rados.0.trial068.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2026-02-25T10:18:07.189 INFO:tasks.rados.rados.0.trial068.stdout:48: writing trial06842131-48 from 789929 to 1424750 tid 1 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:48: writing trial06842131-48 from 2050536 to 2789929 tid 2 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:41: finishing write tid 3 to trial06842131-41 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:31: finishing write tid 3 to trial06842131-31 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:41: finishing write tid 4 to trial06842131-41 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:41: oid 41 updating version 0 to 9 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:41: oid 41 version 9 is already newer than 7 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:41: oid 41 version 9 is already newer than 9 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:36: finishing write tid 1 to trial06842131-36 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:31: finishing write tid 4 to trial06842131-31 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:31: oid 31 updating version 0 to 4 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:31: oid 31 updating version 4 to 6 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:31: oid 31 version 6 is already newer than 6 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:28: finishing write tid 2 to trial06842131-28 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:31: done (15 left) 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:41: done (14 left) 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:49: write initial oid 49 2026-02-25T10:18:07.195 INFO:tasks.rados.rados.0.trial068.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2026-02-25T10:18:07.199 INFO:tasks.rados.rados.0.trial068.stdout:49: writing trial06842131-49 from 406736 to 1116806 tid 1 2026-02-25T10:18:07.204 INFO:tasks.rados.rados.0.trial068.stdout:49: writing trial06842131-49 from 1709018 to 2364368 tid 2 2026-02-25T10:18:07.204 INFO:tasks.rados.rados.0.trial068.stdout:50: write initial oid 50 2026-02-25T10:18:07.204 INFO:tasks.rados.rados.0.trial068.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2026-02-25T10:18:07.208 INFO:tasks.rados.rados.0.trial068.stdout:50: writing trial06842131-50 from 423543 to 1208862 tid 1 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:50: writing trial06842131-50 from 1767500 to 2566508 tid 2 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:40: finishing write tid 1 to trial06842131-40 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:46: finishing write tid 1 to trial06842131-46 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:25: finishing write tid 1 to trial06842131-25 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:28: finishing write tid 3 to trial06842131-28 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:28: finishing write tid 4 to trial06842131-28 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:28: oid 28 updating version 0 to 7 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:28: oid 28 updating version 7 to 9 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:28: oid 28 version 9 is already newer than 9 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:36: finishing write tid 2 to trial06842131-36 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:46: finishing write tid 2 to trial06842131-46 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:28: done (15 left) 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:51: write initial oid 51 2026-02-25T10:18:07.214 INFO:tasks.rados.rados.0.trial068.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2026-02-25T10:18:07.217 INFO:tasks.rados.rados.0.trial068.stdout:51: writing trial06842131-51 from 440350 to 900918 tid 1 2026-02-25T10:18:07.221 INFO:tasks.rados.rados.0.trial068.stdout:51: writing trial06842131-51 from 1309629 to 1852295 tid 2 2026-02-25T10:18:07.225 INFO:tasks.rados.rados.0.trial068.stdout:51: writing trial06842131-51 from 2323973 to 2840350 tid 3 2026-02-25T10:18:07.225 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.225 INFO:tasks.rados.rados.0.trial068.stdout:39: finishing write tid 2 to trial06842131-39 2026-02-25T10:18:07.225 INFO:tasks.rados.rados.0.trial068.stdout:43: finishing write tid 1 to trial06842131-43 2026-02-25T10:18:07.229 INFO:tasks.rados.rados.0.trial068.stdout:25: finishing write tid 2 to trial06842131-25 2026-02-25T10:18:07.232 INFO:tasks.rados.rados.0.trial068.stdout:46: finishing write tid 3 to trial06842131-46 2026-02-25T10:18:07.238 INFO:tasks.rados.rados.0.trial068.stdout:46: finishing write tid 4 to trial06842131-46 2026-02-25T10:18:07.238 INFO:tasks.rados.rados.0.trial068.stdout:46: oid 46 updating version 0 to 9 2026-02-25T10:18:07.238 INFO:tasks.rados.rados.0.trial068.stdout:46: oid 46 version 9 is already newer than 7 2026-02-25T10:18:07.238 INFO:tasks.rados.rados.0.trial068.stdout:46: oid 46 version 9 is already newer than 9 2026-02-25T10:18:07.238 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2026-02-25T10:18:07.239 INFO:tasks.rados.rados.0.trial068.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2026-02-25T10:18:07.239 INFO:tasks.rados.rados.0.trial068.stdout:46: done (15 left) 2026-02-25T10:18:07.239 INFO:tasks.rados.rados.0.trial068.stdout:52: write initial oid 52 2026-02-25T10:18:07.239 INFO:tasks.rados.rados.0.trial068.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2026-02-25T10:18:07.242 INFO:tasks.rados.rados.0.trial068.stdout:52: writing trial06842131-52 from 457157 to 992974 tid 1 2026-02-25T10:18:07.247 INFO:tasks.rados.rados.0.trial068.stdout:52: writing trial06842131-52 from 1768111 to 2170788 tid 2 2026-02-25T10:18:07.247 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.247 INFO:tasks.rados.rados.0.trial068.stdout:32: finishing write tid 1 to trial06842131-32 2026-02-25T10:18:07.247 INFO:tasks.rados.rados.0.trial068.stdout:36: finishing write tid 3 to trial06842131-36 2026-02-25T10:18:07.247 INFO:tasks.rados.rados.0.trial068.stdout:39: finishing write tid 3 to trial06842131-39 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:36: finishing write tid 4 to trial06842131-36 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:36: oid 36 updating version 0 to 9 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:36: oid 36 version 9 is already newer than 9 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:36: oid 36 version 9 is already newer than 7 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:36: done (15 left) 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:53: write initial oid 53 2026-02-25T10:18:07.248 INFO:tasks.rados.rados.0.trial068.stdout:53: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2026-02-25T10:18:07.251 INFO:tasks.rados.rados.0.trial068.stdout:53: writing trial06842131-53 from 473964 to 1085030 tid 1 2026-02-25T10:18:07.256 INFO:tasks.rados.rados.0.trial068.stdout:53: writing trial06842131-53 from 1826593 to 2372928 tid 2 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:53: writing trial06842131-53 from 2778819 to 2873964 tid 3 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:39: finishing write tid 4 to trial06842131-39 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:39: oid 39 updating version 0 to 9 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:39: oid 39 version 9 is already newer than 7 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:39: oid 39 version 9 is already newer than 9 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:39: done (15 left) 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:54: write initial oid 54 2026-02-25T10:18:07.259 INFO:tasks.rados.rados.0.trial068.stdout:54: seq_num 53 ranges {490771=686315,1885075=406346} 2026-02-25T10:18:07.263 INFO:tasks.rados.rados.0.trial068.stdout:54: writing trial06842131-54 from 490771 to 1177086 tid 1 2026-02-25T10:18:07.267 INFO:tasks.rados.rados.0.trial068.stdout:54: writing trial06842131-54 from 1885075 to 2291421 tid 2 2026-02-25T10:18:07.267 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:25: finishing write tid 3 to trial06842131-25 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:43: finishing write tid 2 to trial06842131-43 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:25: finishing write tid 4 to trial06842131-25 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:25: oid 25 updating version 0 to 12 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:25: oid 25 version 12 is already newer than 12 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:25: oid 25 version 12 is already newer than 10 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:25: done (15 left) 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:55: write initial oid 55 2026-02-25T10:18:07.268 INFO:tasks.rados.rados.0.trial068.stdout:55: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2026-02-25T10:18:07.271 INFO:tasks.rados.rados.0.trial068.stdout:55: writing trial06842131-55 from 507578 to 985495 tid 1 2026-02-25T10:18:07.275 INFO:tasks.rados.rados.0.trial068.stdout:55: writing trial06842131-55 from 1543557 to 2093561 tid 2 2026-02-25T10:18:07.278 INFO:tasks.rados.rados.0.trial068.stdout:55: writing trial06842131-55 from 2833665 to 2907578 tid 3 2026-02-25T10:18:07.278 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.279 INFO:tasks.rados.rados.0.trial068.stdout:33: finishing write tid 1 to trial06842131-33 2026-02-25T10:18:07.279 INFO:tasks.rados.rados.0.trial068.stdout:40: finishing write tid 2 to trial06842131-40 2026-02-25T10:18:07.279 INFO:tasks.rados.rados.0.trial068.stdout:47: finishing write tid 1 to trial06842131-47 2026-02-25T10:18:07.279 INFO:tasks.rados.rados.0.trial068.stdout:43: finishing write tid 3 to trial06842131-43 2026-02-25T10:18:07.281 INFO:tasks.rados.rados.0.trial068.stdout:43: finishing write tid 4 to trial06842131-43 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:43: oid 43 updating version 0 to 10 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:43: oid 43 updating version 10 to 12 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:43: oid 43 version 12 is already newer than 12 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:43: done (15 left) 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:56: write initial oid 56 2026-02-25T10:18:07.282 INFO:tasks.rados.rados.0.trial068.stdout:56: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2026-02-25T10:18:07.285 INFO:tasks.rados.rados.0.trial068.stdout:56: writing trial06842131-56 from 524385 to 1077551 tid 1 2026-02-25T10:18:07.289 INFO:tasks.rados.rados.0.trial068.stdout:56: writing trial06842131-56 from 1602039 to 2012054 tid 2 2026-02-25T10:18:07.293 INFO:tasks.rados.rados.0.trial068.stdout:56: writing trial06842131-56 from 2577441 to 2924385 tid 3 2026-02-25T10:18:07.293 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:33: finishing write tid 2 to trial06842131-33 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:40: finishing write tid 3 to trial06842131-40 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:32: finishing write tid 2 to trial06842131-32 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:40: finishing write tid 4 to trial06842131-40 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:40: oid 40 updating version 0 to 9 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:40: oid 40 version 9 is already newer than 7 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:40: oid 40 version 9 is already newer than 9 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:40: done (15 left) 2026-02-25T10:18:07.294 INFO:tasks.rados.rados.0.trial068.stdout:57: write initial oid 57 2026-02-25T10:18:07.295 INFO:tasks.rados.rados.0.trial068.stdout:57: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2026-02-25T10:18:07.297 INFO:tasks.rados.rados.0.trial068.stdout:57: writing trial06842131-57 from 541192 to 1169607 tid 1 2026-02-25T10:18:07.301 INFO:tasks.rados.rados.0.trial068.stdout:57: writing trial06842131-57 from 1660521 to 2214194 tid 2 2026-02-25T10:18:07.305 INFO:tasks.rados.rados.0.trial068.stdout:57: writing trial06842131-57 from 2888511 to 2941192 tid 3 2026-02-25T10:18:07.305 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.305 INFO:tasks.rados.rados.0.trial068.stdout:33: finishing write tid 3 to trial06842131-33 2026-02-25T10:18:07.305 INFO:tasks.rados.rados.0.trial068.stdout:33: finishing write tid 4 to trial06842131-33 2026-02-25T10:18:07.305 INFO:tasks.rados.rados.0.trial068.stdout:33: oid 33 updating version 0 to 13 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:33: oid 33 updating version 13 to 15 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:33: oid 33 version 15 is already newer than 15 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 33 v 15 (ObjNum 32 snap 0 seq_num 32) dirty exists 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:32: finishing write tid 3 to trial06842131-32 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:32: finishing write tid 4 to trial06842131-32 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:32: oid 32 updating version 0 to 7 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:32: oid 32 updating version 7 to 9 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:32: oid 32 version 9 is already newer than 9 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2026-02-25T10:18:07.306 INFO:tasks.rados.rados.0.trial068.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2026-02-25T10:18:07.307 INFO:tasks.rados.rados.0.trial068.stdout:44: finishing write tid 1 to trial06842131-44 2026-02-25T10:18:07.307 INFO:tasks.rados.rados.0.trial068.stdout:32: done (15 left) 2026-02-25T10:18:07.307 INFO:tasks.rados.rados.0.trial068.stdout:33: done (14 left) 2026-02-25T10:18:07.307 INFO:tasks.rados.rados.0.trial068.stdout:58: write initial oid 58 2026-02-25T10:18:07.307 INFO:tasks.rados.rados.0.trial068.stdout:58: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2026-02-25T10:18:07.309 INFO:tasks.rados.rados.0.trial068.stdout:58: writing trial06842131-58 from 557999 to 1261663 tid 1 2026-02-25T10:18:07.312 INFO:tasks.rados.rados.0.trial068.stdout:58: writing trial06842131-58 from 1719003 to 2132687 tid 2 2026-02-25T10:18:07.316 INFO:tasks.rados.rados.0.trial068.stdout:58: writing trial06842131-58 from 2632287 to 2957999 tid 3 2026-02-25T10:18:07.316 INFO:tasks.rados.rados.0.trial068.stdout:59: write initial oid 59 2026-02-25T10:18:07.316 INFO:tasks.rados.rados.0.trial068.stdout:59: seq_num 58 ranges {574806=778913,2061132=557342} 2026-02-25T10:18:07.321 INFO:tasks.rados.rados.0.trial068.stdout:59: writing trial06842131-59 from 574806 to 1353719 tid 1 2026-02-25T10:18:07.326 INFO:tasks.rados.rados.0.trial068.stdout:59: writing trial06842131-59 from 2061132 to 2618474 tid 2 2026-02-25T10:18:07.326 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.326 INFO:tasks.rados.rados.0.trial068.stdout:54: finishing write tid 1 to trial06842131-54 2026-02-25T10:18:07.326 INFO:tasks.rados.rados.0.trial068.stdout:45: finishing write tid 1 to trial06842131-45 2026-02-25T10:18:07.326 INFO:tasks.rados.rados.0.trial068.stdout:54: finishing write tid 2 to trial06842131-54 2026-02-25T10:18:07.326 INFO:tasks.rados.rados.0.trial068.stdout:47: finishing write tid 2 to trial06842131-47 2026-02-25T10:18:07.330 INFO:tasks.rados.rados.0.trial068.stdout:54: finishing write tid 3 to trial06842131-54 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:54: finishing write tid 4 to trial06842131-54 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:54: oid 54 updating version 0 to 10 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:54: oid 54 updating version 10 to 12 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:54: oid 54 version 12 is already newer than 12 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 54 v 12 (ObjNum 53 snap 0 seq_num 53) dirty exists 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:54: left oid 54 (ObjNum 53 snap 0 seq_num 53) 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:54: done (15 left) 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:60: write initial oid 60 2026-02-25T10:18:07.334 INFO:tasks.rados.rados.0.trial068.stdout:60: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2026-02-25T10:18:07.337 INFO:tasks.rados.rados.0.trial068.stdout:60: writing trial06842131-60 from 591613 to 1045775 tid 1 2026-02-25T10:18:07.341 INFO:tasks.rados.rados.0.trial068.stdout:60: writing trial06842131-60 from 1719614 to 2136967 tid 2 2026-02-25T10:18:07.345 INFO:tasks.rados.rados.0.trial068.stdout:60: writing trial06842131-60 from 2570780 to 2991613 tid 3 2026-02-25T10:18:07.345 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.345 INFO:tasks.rados.rados.0.trial068.stdout:51: finishing write tid 1 to trial06842131-51 2026-02-25T10:18:07.345 INFO:tasks.rados.rados.0.trial068.stdout:50: finishing write tid 1 to trial06842131-50 2026-02-25T10:18:07.346 INFO:tasks.rados.rados.0.trial068.stdout:44: finishing write tid 2 to trial06842131-44 2026-02-25T10:18:07.349 INFO:tasks.rados.rados.0.trial068.stdout:45: finishing write tid 2 to trial06842131-45 2026-02-25T10:18:07.353 INFO:tasks.rados.rados.0.trial068.stdout:50: finishing write tid 2 to trial06842131-50 2026-02-25T10:18:07.364 INFO:tasks.rados.rados.0.trial068.stdout:45: finishing write tid 3 to trial06842131-45 2026-02-25T10:18:07.365 INFO:tasks.rados.rados.0.trial068.stdout:47: finishing write tid 3 to trial06842131-47 2026-02-25T10:18:07.369 INFO:tasks.rados.rados.0.trial068.stdout:45: finishing write tid 4 to trial06842131-45 2026-02-25T10:18:07.369 INFO:tasks.rados.rados.0.trial068.stdout:45: oid 45 updating version 0 to 9 2026-02-25T10:18:07.369 INFO:tasks.rados.rados.0.trial068.stdout:45: oid 45 version 9 is already newer than 7 2026-02-25T10:18:07.369 INFO:tasks.rados.rados.0.trial068.stdout:45: oid 45 version 9 is already newer than 9 2026-02-25T10:18:07.370 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2026-02-25T10:18:07.370 INFO:tasks.rados.rados.0.trial068.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2026-02-25T10:18:07.370 INFO:tasks.rados.rados.0.trial068.stdout:45: done (15 left) 2026-02-25T10:18:07.370 INFO:tasks.rados.rados.0.trial068.stdout:61: write initial oid 61 2026-02-25T10:18:07.370 INFO:tasks.rados.rados.0.trial068.stdout:61: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2026-02-25T10:18:07.373 INFO:tasks.rados.rados.0.trial068.stdout:61: writing trial06842131-61 from 608420 to 1137831 tid 1 2026-02-25T10:18:07.377 INFO:tasks.rados.rados.0.trial068.stdout:61: writing trial06842131-61 from 1778096 to 2339107 tid 2 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:61: writing trial06842131-61 from 2881850 to 3008420 tid 3 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:47: finishing write tid 4 to trial06842131-47 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:47: oid 47 updating version 0 to 9 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:47: oid 47 version 9 is already newer than 9 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:47: oid 47 version 9 is already newer than 7 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2026-02-25T10:18:07.381 INFO:tasks.rados.rados.0.trial068.stdout:51: finishing write tid 2 to trial06842131-51 2026-02-25T10:18:07.382 INFO:tasks.rados.rados.0.trial068.stdout:47: done (15 left) 2026-02-25T10:18:07.382 INFO:tasks.rados.rados.0.trial068.stdout:62: write initial oid 62 2026-02-25T10:18:07.382 INFO:tasks.rados.rados.0.trial068.stdout:62: seq_num 61 ranges {625227=721013,1952931=704669} 2026-02-25T10:18:07.385 INFO:tasks.rados.rados.0.trial068.stdout:62: writing trial06842131-62 from 625227 to 1346240 tid 1 2026-02-25T10:18:07.390 INFO:tasks.rados.rados.0.trial068.stdout:62: writing trial06842131-62 from 1952931 to 2657600 tid 2 2026-02-25T10:18:07.390 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.391 INFO:tasks.rados.rados.0.trial068.stdout:52: finishing write tid 1 to trial06842131-52 2026-02-25T10:18:07.391 INFO:tasks.rados.rados.0.trial068.stdout:50: finishing write tid 3 to trial06842131-50 2026-02-25T10:18:07.391 INFO:tasks.rados.rados.0.trial068.stdout:53: finishing write tid 1 to trial06842131-53 2026-02-25T10:18:07.391 INFO:tasks.rados.rados.0.trial068.stdout:42: finishing write tid 1 to trial06842131-42 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:50: finishing write tid 4 to trial06842131-50 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:50: oid 50 updating version 0 to 10 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:50: oid 50 updating version 10 to 12 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:50: oid 50 version 12 is already newer than 12 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:50: done (15 left) 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:63: write initial oid 63 2026-02-25T10:18:07.392 INFO:tasks.rados.rados.0.trial068.stdout:63: seq_num 62 ranges {642034=796262,1895060=564680} 2026-02-25T10:18:07.396 INFO:tasks.rados.rados.0.trial068.stdout:63: writing trial06842131-63 from 642034 to 1438296 tid 1 2026-02-25T10:18:07.400 INFO:tasks.rados.rados.0.trial068.stdout:63: writing trial06842131-63 from 1895060 to 2459740 tid 2 2026-02-25T10:18:07.401 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.401 INFO:tasks.rados.rados.0.trial068.stdout:51: finishing write tid 3 to trial06842131-51 2026-02-25T10:18:07.408 INFO:tasks.rados.rados.0.trial068.stdout:49: finishing write tid 1 to trial06842131-49 2026-02-25T10:18:07.416 INFO:tasks.rados.rados.0.trial068.stdout:51: finishing write tid 4 to trial06842131-51 2026-02-25T10:18:07.420 INFO:tasks.rados.rados.0.trial068.stdout:51: finishing write tid 5 to trial06842131-51 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:51: oid 51 updating version 0 to 13 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:51: oid 51 updating version 13 to 16 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:51: oid 51 version 16 is already newer than 16 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:51: oid 51 version 16 is already newer than 16 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 51 v 16 (ObjNum 50 snap 0 seq_num 50) dirty exists 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:51: left oid 51 (ObjNum 50 snap 0 seq_num 50) 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:51: done (15 left) 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:64: write initial oid 64 2026-02-25T10:18:07.421 INFO:tasks.rados.rados.0.trial068.stdout:64: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2026-02-25T10:18:07.425 INFO:tasks.rados.rados.0.trial068.stdout:64: writing trial06842131-64 from 658841 to 1130352 tid 1 2026-02-25T10:18:07.429 INFO:tasks.rados.rados.0.trial068.stdout:64: writing trial06842131-64 from 1553542 to 2261880 tid 2 2026-02-25T10:18:07.432 INFO:tasks.rados.rados.0.trial068.stdout:64: writing trial06842131-64 from 2964119 to 3058841 tid 3 2026-02-25T10:18:07.432 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.433 INFO:tasks.rados.rados.0.trial068.stdout:58: finishing write tid 1 to trial06842131-58 2026-02-25T10:18:07.435 INFO:tasks.rados.rados.0.trial068.stdout:53: finishing write tid 2 to trial06842131-53 2026-02-25T10:18:07.445 INFO:tasks.rados.rados.0.trial068.stdout:42: finishing write tid 2 to trial06842131-42 2026-02-25T10:18:07.449 INFO:tasks.rados.rados.0.trial068.stdout:49: finishing write tid 2 to trial06842131-49 2026-02-25T10:18:07.459 INFO:tasks.rados.rados.0.trial068.stdout:59: finishing write tid 1 to trial06842131-59 2026-02-25T10:18:07.465 INFO:tasks.rados.rados.0.trial068.stdout:49: finishing write tid 3 to trial06842131-49 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:49: finishing write tid 4 to trial06842131-49 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:49: oid 49 updating version 0 to 10 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:49: oid 49 updating version 10 to 12 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:49: oid 49 version 12 is already newer than 12 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:49: done (15 left) 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:65: write initial oid 65 2026-02-25T10:18:07.471 INFO:tasks.rados.rados.0.trial068.stdout:65: seq_num 64 ranges {675648=546760,2012024=568349} 2026-02-25T10:18:07.475 INFO:tasks.rados.rados.0.trial068.stdout:65: writing trial06842131-65 from 675648 to 1222408 tid 1 2026-02-25T10:18:07.480 INFO:tasks.rados.rados.0.trial068.stdout:65: writing trial06842131-65 from 2012024 to 2580373 tid 2 2026-02-25T10:18:07.480 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.480 INFO:tasks.rados.rados.0.trial068.stdout:59: finishing write tid 2 to trial06842131-59 2026-02-25T10:18:07.481 INFO:tasks.rados.rados.0.trial068.stdout:44: finishing write tid 3 to trial06842131-44 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:44: finishing write tid 4 to trial06842131-44 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:44: oid 44 updating version 0 to 12 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:44: oid 44 version 12 is already newer than 10 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:44: oid 44 version 12 is already newer than 12 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:44: done (15 left) 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:66: write initial oid 66 2026-02-25T10:18:07.485 INFO:tasks.rados.rados.0.trial068.stdout:66: seq_num 65 ranges {692455=622009,2070506=712007} 2026-02-25T10:18:07.490 INFO:tasks.rados.rados.0.trial068.stdout:66: writing trial06842131-66 from 692455 to 1314464 tid 1 2026-02-25T10:18:07.496 INFO:tasks.rados.rados.0.trial068.stdout:66: writing trial06842131-66 from 2070506 to 2782513 tid 2 2026-02-25T10:18:07.496 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.496 INFO:tasks.rados.rados.0.trial068.stdout:65: finishing write tid 1 to trial06842131-65 2026-02-25T10:18:07.496 INFO:tasks.rados.rados.0.trial068.stdout:53: finishing write tid 3 to trial06842131-53 2026-02-25T10:18:07.496 INFO:tasks.rados.rados.0.trial068.stdout:52: finishing write tid 2 to trial06842131-52 2026-02-25T10:18:07.509 INFO:tasks.rados.rados.0.trial068.stdout:53: finishing write tid 4 to trial06842131-53 2026-02-25T10:18:07.510 INFO:tasks.rados.rados.0.trial068.stdout:65: finishing write tid 2 to trial06842131-65 2026-02-25T10:18:07.510 INFO:tasks.rados.rados.0.trial068.stdout:58: finishing write tid 2 to trial06842131-58 2026-02-25T10:18:07.512 INFO:tasks.rados.rados.0.trial068.stdout:53: finishing write tid 5 to trial06842131-53 2026-02-25T10:18:07.512 INFO:tasks.rados.rados.0.trial068.stdout:53: oid 53 updating version 0 to 13 2026-02-25T10:18:07.512 INFO:tasks.rados.rados.0.trial068.stdout:53: oid 53 version 13 is already newer than 10 2026-02-25T10:18:07.512 INFO:tasks.rados.rados.0.trial068.stdout:53: oid 53 version 13 is already newer than 13 2026-02-25T10:18:07.512 INFO:tasks.rados.rados.0.trial068.stdout:53: oid 53 version 13 is already newer than 13 2026-02-25T10:18:07.512 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 53 v 13 (ObjNum 52 snap 0 seq_num 52) dirty exists 2026-02-25T10:18:07.513 INFO:tasks.rados.rados.0.trial068.stdout:53: left oid 53 (ObjNum 52 snap 0 seq_num 52) 2026-02-25T10:18:07.513 INFO:tasks.rados.rados.0.trial068.stdout:53: done (15 left) 2026-02-25T10:18:07.513 INFO:tasks.rados.rados.0.trial068.stdout:67: write initial oid 67 2026-02-25T10:18:07.513 INFO:tasks.rados.rados.0.trial068.stdout:67: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2026-02-25T10:18:07.517 INFO:tasks.rados.rados.0.trial068.stdout:67: writing trial06842131-67 from 709262 to 1406520 tid 1 2026-02-25T10:18:07.522 INFO:tasks.rados.rados.0.trial068.stdout:67: writing trial06842131-67 from 2012635 to 2584653 tid 2 2026-02-25T10:18:07.524 INFO:tasks.rados.rados.0.trial068.stdout:67: writing trial06842131-67 from 3046388 to 3109262 tid 3 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:66: finishing write tid 1 to trial06842131-66 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:65: finishing write tid 3 to trial06842131-65 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:65: finishing write tid 4 to trial06842131-65 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:65: oid 65 updating version 0 to 10 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:65: oid 65 updating version 10 to 12 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:65: oid 65 version 12 is already newer than 12 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 65 v 12 (ObjNum 64 snap 0 seq_num 64) dirty exists 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:65: left oid 65 (ObjNum 64 snap 0 seq_num 64) 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:55: finishing write tid 1 to trial06842131-55 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:65: done (15 left) 2026-02-25T10:18:07.525 INFO:tasks.rados.rados.0.trial068.stdout:68: write initial oid 68 2026-02-25T10:18:07.526 INFO:tasks.rados.rados.0.trial068.stdout:68: seq_num 67 ranges {726069=772507,2071117=715676} 2026-02-25T10:18:07.530 INFO:tasks.rados.rados.0.trial068.stdout:68: writing trial06842131-68 from 726069 to 1498576 tid 1 2026-02-25T10:18:07.535 INFO:tasks.rados.rados.0.trial068.stdout:68: writing trial06842131-68 from 2071117 to 2786793 tid 2 2026-02-25T10:18:07.535 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.535 INFO:tasks.rados.rados.0.trial068.stdout:52: finishing write tid 3 to trial06842131-52 2026-02-25T10:18:07.535 INFO:tasks.rados.rados.0.trial068.stdout:52: finishing write tid 4 to trial06842131-52 2026-02-25T10:18:07.535 INFO:tasks.rados.rados.0.trial068.stdout:52: oid 52 updating version 0 to 12 2026-02-25T10:18:07.536 INFO:tasks.rados.rados.0.trial068.stdout:52: oid 52 version 12 is already newer than 10 2026-02-25T10:18:07.536 INFO:tasks.rados.rados.0.trial068.stdout:52: oid 52 version 12 is already newer than 12 2026-02-25T10:18:07.536 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 52 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2026-02-25T10:18:07.536 INFO:tasks.rados.rados.0.trial068.stdout:52: left oid 52 (ObjNum 51 snap 0 seq_num 51) 2026-02-25T10:18:07.536 INFO:tasks.rados.rados.0.trial068.stdout:52: done (15 left) 2026-02-25T10:18:07.536 INFO:tasks.rados.rados.0.trial068.stdout:69: write initial oid 69 2026-02-25T10:18:07.536 INFO:tasks.rados.rados.0.trial068.stdout:69: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2026-02-25T10:18:07.539 INFO:tasks.rados.rados.0.trial068.stdout:69: writing trial06842131-69 from 742876 to 1190632 tid 1 2026-02-25T10:18:07.543 INFO:tasks.rados.rados.0.trial068.stdout:69: writing trial06842131-69 from 1729599 to 2305286 tid 2 2026-02-25T10:18:07.547 INFO:tasks.rados.rados.0.trial068.stdout:69: writing trial06842131-69 from 3101234 to 3142876 tid 3 2026-02-25T10:18:07.547 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.547 INFO:tasks.rados.rados.0.trial068.stdout:66: finishing write tid 2 to trial06842131-66 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:42: finishing write tid 3 to trial06842131-42 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:42: finishing write tid 4 to trial06842131-42 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:42: oid 42 updating version 0 to 12 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:42: oid 42 version 12 is already newer than 12 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:42: oid 42 version 12 is already newer than 10 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:57: finishing write tid 1 to trial06842131-57 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:55: finishing write tid 2 to trial06842131-55 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:42: done (15 left) 2026-02-25T10:18:07.548 INFO:tasks.rados.rados.0.trial068.stdout:70: write initial oid 70 2026-02-25T10:18:07.549 INFO:tasks.rados.rados.0.trial068.stdout:70: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2026-02-25T10:18:07.552 INFO:tasks.rados.rados.0.trial068.stdout:70: writing trial06842131-70 from 759683 to 1399041 tid 1 2026-02-25T10:18:07.557 INFO:tasks.rados.rados.0.trial068.stdout:70: writing trial06842131-70 from 1904434 to 2623779 tid 2 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:70: writing trial06842131-70 from 3128657 to 3159683 tid 3 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:68: finishing write tid 1 to trial06842131-68 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:58: finishing write tid 3 to trial06842131-58 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:59: finishing write tid 3 to trial06842131-59 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:59: finishing write tid 4 to trial06842131-59 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:59: oid 59 updating version 0 to 19 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:59: oid 59 version 19 is already newer than 17 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:59: oid 59 version 19 is already newer than 19 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 59 v 19 (ObjNum 58 snap 0 seq_num 58) dirty exists 2026-02-25T10:18:07.560 INFO:tasks.rados.rados.0.trial068.stdout:59: left oid 59 (ObjNum 58 snap 0 seq_num 58) 2026-02-25T10:18:07.561 INFO:tasks.rados.rados.0.trial068.stdout:59: done (15 left) 2026-02-25T10:18:07.561 INFO:tasks.rados.rados.0.trial068.stdout:71: write initial oid 71 2026-02-25T10:18:07.561 INFO:tasks.rados.rados.0.trial068.stdout:71: seq_num 70 ranges {776490=714607,2246563=579356} 2026-02-25T10:18:07.565 INFO:tasks.rados.rados.0.trial068.stdout:71: writing trial06842131-71 from 776490 to 1491097 tid 1 2026-02-25T10:18:07.570 INFO:tasks.rados.rados.0.trial068.stdout:71: writing trial06842131-71 from 2246563 to 2825919 tid 2 2026-02-25T10:18:07.570 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.570 INFO:tasks.rados.rados.0.trial068.stdout:58: finishing write tid 4 to trial06842131-58 2026-02-25T10:18:07.570 INFO:tasks.rados.rados.0.trial068.stdout:55: finishing write tid 3 to trial06842131-55 2026-02-25T10:18:07.570 INFO:tasks.rados.rados.0.trial068.stdout:68: finishing write tid 2 to trial06842131-68 2026-02-25T10:18:07.570 INFO:tasks.rados.rados.0.trial068.stdout:67: finishing write tid 1 to trial06842131-67 2026-02-25T10:18:07.571 INFO:tasks.rados.rados.0.trial068.stdout:58: finishing write tid 5 to trial06842131-58 2026-02-25T10:18:07.571 INFO:tasks.rados.rados.0.trial068.stdout:58: oid 58 updating version 0 to 13 2026-02-25T10:18:07.571 INFO:tasks.rados.rados.0.trial068.stdout:58: oid 58 version 13 is already newer than 13 2026-02-25T10:18:07.571 INFO:tasks.rados.rados.0.trial068.stdout:58: oid 58 version 13 is already newer than 13 2026-02-25T10:18:07.572 INFO:tasks.rados.rados.0.trial068.stdout:58: oid 58 version 13 is already newer than 10 2026-02-25T10:18:07.572 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 58 v 13 (ObjNum 57 snap 0 seq_num 57) dirty exists 2026-02-25T10:18:07.572 INFO:tasks.rados.rados.0.trial068.stdout:58: left oid 58 (ObjNum 57 snap 0 seq_num 57) 2026-02-25T10:18:07.572 INFO:tasks.rados.rados.0.trial068.stdout:58: done (15 left) 2026-02-25T10:18:07.572 INFO:tasks.rados.rados.0.trial068.stdout:72: write initial oid 72 2026-02-25T10:18:07.572 INFO:tasks.rados.rados.0.trial068.stdout:72: seq_num 71 ranges {793297=789856,2305045=723014} 2026-02-25T10:18:07.577 INFO:tasks.rados.rados.0.trial068.stdout:72: writing trial06842131-72 from 793297 to 1583153 tid 1 2026-02-25T10:18:07.584 INFO:tasks.rados.rados.0.trial068.stdout:72: writing trial06842131-72 from 2305045 to 3028059 tid 2 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:68: finishing write tid 3 to trial06842131-68 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:68: finishing write tid 4 to trial06842131-68 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:68: oid 68 updating version 0 to 6 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:68: oid 68 version 6 is already newer than 6 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:68: oid 68 version 6 is already newer than 4 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 68 v 6 (ObjNum 67 snap 0 seq_num 67) dirty exists 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:68: left oid 68 (ObjNum 67 snap 0 seq_num 67) 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:66: finishing write tid 3 to trial06842131-66 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:66: finishing write tid 4 to trial06842131-66 2026-02-25T10:18:07.585 INFO:tasks.rados.rados.0.trial068.stdout:66: oid 66 updating version 0 to 15 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:66: oid 66 version 15 is already newer than 15 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:66: oid 66 version 15 is already newer than 13 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 66 v 15 (ObjNum 65 snap 0 seq_num 65) dirty exists 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:66: left oid 66 (ObjNum 65 snap 0 seq_num 65) 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:55: finishing write tid 4 to trial06842131-55 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:66: done (15 left) 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:68: done (14 left) 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:73: write initial oid 73 2026-02-25T10:18:07.586 INFO:tasks.rados.rados.0.trial068.stdout:73: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2026-02-25T10:18:07.587 INFO:tasks.rados.rados.0.trial068.stdout:73: writing trial06842131-73 from 410104 to 875209 tid 1 2026-02-25T10:18:07.592 INFO:tasks.rados.rados.0.trial068.stdout:73: writing trial06842131-73 from 1563527 to 2146552 tid 2 2026-02-25T10:18:07.596 INFO:tasks.rados.rados.0.trial068.stdout:73: writing trial06842131-73 from 2810926 to 3210104 tid 3 2026-02-25T10:18:07.596 INFO:tasks.rados.rados.0.trial068.stdout:74: write initial oid 74 2026-02-25T10:18:07.596 INFO:tasks.rados.rados.0.trial068.stdout:74: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2026-02-25T10:18:07.600 INFO:tasks.rados.rados.0.trial068.stdout:74: writing trial06842131-74 from 426911 to 967265 tid 1 2026-02-25T10:18:07.605 INFO:tasks.rados.rados.0.trial068.stdout:74: writing trial06842131-74 from 1622009 to 2348692 tid 2 2026-02-25T10:18:07.609 INFO:tasks.rados.rados.0.trial068.stdout:74: writing trial06842131-74 from 3121996 to 3226911 tid 3 2026-02-25T10:18:07.609 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.609 INFO:tasks.rados.rados.0.trial068.stdout:55: finishing write tid 5 to trial06842131-55 2026-02-25T10:18:07.609 INFO:tasks.rados.rados.0.trial068.stdout:55: oid 55 updating version 0 to 17 2026-02-25T10:18:07.609 INFO:tasks.rados.rados.0.trial068.stdout:55: oid 55 version 17 is already newer than 17 2026-02-25T10:18:07.609 INFO:tasks.rados.rados.0.trial068.stdout:55: oid 55 version 17 is already newer than 17 2026-02-25T10:18:07.609 INFO:tasks.rados.rados.0.trial068.stdout:55: oid 55 version 17 is already newer than 14 2026-02-25T10:18:07.610 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 55 v 17 (ObjNum 54 snap 0 seq_num 54) dirty exists 2026-02-25T10:18:07.610 INFO:tasks.rados.rados.0.trial068.stdout:55: left oid 55 (ObjNum 54 snap 0 seq_num 54) 2026-02-25T10:18:07.610 INFO:tasks.rados.rados.0.trial068.stdout:57: finishing write tid 2 to trial06842131-57 2026-02-25T10:18:07.610 INFO:tasks.rados.rados.0.trial068.stdout:55: done (15 left) 2026-02-25T10:18:07.610 INFO:tasks.rados.rados.0.trial068.stdout:75: write initial oid 75 2026-02-25T10:18:07.610 INFO:tasks.rados.rados.0.trial068.stdout:75: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2026-02-25T10:18:07.612 INFO:tasks.rados.rados.0.trial068.stdout:75: writing trial06842131-75 from 443718 to 1059321 tid 1 2026-02-25T10:18:07.616 INFO:tasks.rados.rados.0.trial068.stdout:75: writing trial06842131-75 from 1564138 to 2034479 tid 2 2026-02-25T10:18:07.622 INFO:tasks.rados.rados.0.trial068.stdout:75: writing trial06842131-75 from 2633066 to 3243718 tid 3 2026-02-25T10:18:07.622 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.622 INFO:tasks.rados.rados.0.trial068.stdout:63: finishing write tid 1 to trial06842131-63 2026-02-25T10:18:07.622 INFO:tasks.rados.rados.0.trial068.stdout:71: finishing write tid 1 to trial06842131-71 2026-02-25T10:18:07.622 INFO:tasks.rados.rados.0.trial068.stdout:67: finishing write tid 2 to trial06842131-67 2026-02-25T10:18:07.622 INFO:tasks.rados.rados.0.trial068.stdout:57: finishing write tid 3 to trial06842131-57 2026-02-25T10:18:07.622 INFO:tasks.rados.rados.0.trial068.stdout:60: finishing write tid 1 to trial06842131-60 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: finishing write tid 3 to trial06842131-67 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:63: finishing write tid 2 to trial06842131-63 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:57: finishing write tid 4 to trial06842131-57 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: finishing write tid 4 to trial06842131-67 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: finishing write tid 5 to trial06842131-67 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: oid 67 updating version 0 to 19 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: oid 67 version 19 is already newer than 19 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: oid 67 version 19 is already newer than 19 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: oid 67 version 19 is already newer than 16 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 67 v 19 (ObjNum 66 snap 0 seq_num 66) dirty exists 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:67: left oid 67 (ObjNum 66 snap 0 seq_num 66) 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:57: finishing write tid 5 to trial06842131-57 2026-02-25T10:18:07.623 INFO:tasks.rados.rados.0.trial068.stdout:57: oid 57 updating version 0 to 16 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:57: oid 57 version 16 is already newer than 16 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:57: oid 57 version 16 is already newer than 13 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:57: oid 57 version 16 is already newer than 16 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 57 v 16 (ObjNum 56 snap 0 seq_num 56) dirty exists 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:57: left oid 57 (ObjNum 56 snap 0 seq_num 56) 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:57: done (15 left) 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:67: done (14 left) 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:76: write initial oid 76 2026-02-25T10:18:07.624 INFO:tasks.rados.rados.0.trial068.stdout:76: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2026-02-25T10:18:07.625 INFO:tasks.rados.rados.0.trial068.stdout:76: writing trial06842131-76 from 460525 to 1151377 tid 1 2026-02-25T10:18:07.630 INFO:tasks.rados.rados.0.trial068.stdout:76: writing trial06842131-76 from 1622620 to 2352972 tid 2 2026-02-25T10:18:07.633 INFO:tasks.rados.rados.0.trial068.stdout:76: writing trial06842131-76 from 3060489 to 3260525 tid 3 2026-02-25T10:18:07.634 INFO:tasks.rados.rados.0.trial068.stdout:77: write initial oid 77 2026-02-25T10:18:07.634 INFO:tasks.rados.rados.0.trial068.stdout:77: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2026-02-25T10:18:07.638 INFO:tasks.rados.rados.0.trial068.stdout:77: writing trial06842131-77 from 477332 to 1243433 tid 1 2026-02-25T10:18:07.641 INFO:tasks.rados.rados.0.trial068.stdout:77: writing trial06842131-77 from 1681102 to 2155112 tid 2 2026-02-25T10:18:07.647 INFO:tasks.rados.rados.0.trial068.stdout:77: writing trial06842131-77 from 2687912 to 3277332 tid 3 2026-02-25T10:18:07.647 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:60: finishing write tid 2 to trial06842131-60 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:48: finishing write tid 1 to trial06842131-48 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:71: finishing write tid 2 to trial06842131-71 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:71: finishing write tid 3 to trial06842131-71 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:71: finishing write tid 4 to trial06842131-71 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:71: oid 71 updating version 0 to 6 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:71: oid 71 version 6 is already newer than 6 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:71: oid 71 version 6 is already newer than 4 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 71 v 6 (ObjNum 70 snap 0 seq_num 70) dirty exists 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:71: left oid 71 (ObjNum 70 snap 0 seq_num 70) 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:63: finishing write tid 3 to trial06842131-63 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:63: finishing write tid 4 to trial06842131-63 2026-02-25T10:18:07.648 INFO:tasks.rados.rados.0.trial068.stdout:63: oid 63 updating version 0 to 22 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:63: oid 63 version 22 is already newer than 22 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:63: oid 63 version 22 is already newer than 20 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 63 v 22 (ObjNum 62 snap 0 seq_num 62) dirty exists 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:63: left oid 63 (ObjNum 62 snap 0 seq_num 62) 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:72: finishing write tid 1 to trial06842131-72 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:63: done (15 left) 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:71: done (14 left) 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:78: write initial oid 78 2026-02-25T10:18:07.649 INFO:tasks.rados.rados.0.trial068.stdout:78: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2026-02-25T10:18:07.651 INFO:tasks.rados.rados.0.trial068.stdout:78: writing trial06842131-78 from 494139 to 1051842 tid 1 2026-02-25T10:18:07.655 INFO:tasks.rados.rados.0.trial068.stdout:78: writing trial06842131-78 from 1455937 to 2189958 tid 2 2026-02-25T10:18:07.660 INFO:tasks.rados.rados.0.trial068.stdout:78: writing trial06842131-78 from 2948041 to 3294139 tid 3 2026-02-25T10:18:07.660 INFO:tasks.rados.rados.0.trial068.stdout:79: write initial oid 79 2026-02-25T10:18:07.660 INFO:tasks.rados.rados.0.trial068.stdout:79: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2026-02-25T10:18:07.664 INFO:tasks.rados.rados.0.trial068.stdout:79: writing trial06842131-79 from 510946 to 1143898 tid 1 2026-02-25T10:18:07.669 INFO:tasks.rados.rados.0.trial068.stdout:79: writing trial06842131-79 from 1798066 to 2275745 tid 2 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout:79: writing trial06842131-79 from 2742758 to 3199975 tid 3 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout:60: finishing write tid 3 to trial06842131-60 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout:48: finishing write tid 2 to trial06842131-48 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout:60: finishing write tid 4 to trial06842131-60 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout:72: finishing write tid 2 to trial06842131-72 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout:60: finishing write tid 5 to trial06842131-60 2026-02-25T10:18:07.673 INFO:tasks.rados.rados.0.trial068.stdout:60: oid 60 updating version 0 to 21 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:60: oid 60 version 21 is already newer than 18 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:60: oid 60 version 21 is already newer than 21 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:60: oid 60 version 21 is already newer than 21 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 60 v 21 (ObjNum 59 snap 0 seq_num 59) dirty exists 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:60: left oid 60 (ObjNum 59 snap 0 seq_num 59) 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:72: finishing write tid 3 to trial06842131-72 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:60: done (15 left) 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:80: write initial oid 80 2026-02-25T10:18:07.674 INFO:tasks.rados.rados.0.trial068.stdout:80: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2026-02-25T10:18:07.677 INFO:tasks.rados.rados.0.trial068.stdout:80: writing trial06842131-80 from 527753 to 1235954 tid 1 2026-02-25T10:18:07.682 INFO:tasks.rados.rados.0.trial068.stdout:80: writing trial06842131-80 from 1856548 to 2594238 tid 2 2026-02-25T10:18:07.686 INFO:tasks.rados.rados.0.trial068.stdout:80: writing trial06842131-80 from 3286534 to 3327753 tid 3 2026-02-25T10:18:07.686 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.686 INFO:tasks.rados.rados.0.trial068.stdout:72: finishing write tid 4 to trial06842131-72 2026-02-25T10:18:07.686 INFO:tasks.rados.rados.0.trial068.stdout:72: oid 72 updating version 0 to 22 2026-02-25T10:18:07.686 INFO:tasks.rados.rados.0.trial068.stdout:72: oid 72 version 22 is already newer than 22 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:72: oid 72 version 22 is already newer than 20 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 72 v 22 (ObjNum 71 snap 0 seq_num 71) dirty exists 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:72: left oid 72 (ObjNum 71 snap 0 seq_num 71) 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:48: finishing write tid 3 to trial06842131-48 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:74: finishing write tid 1 to trial06842131-74 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:72: done (15 left) 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:81: write initial oid 81 2026-02-25T10:18:07.687 INFO:tasks.rados.rados.0.trial068.stdout:81: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2026-02-25T10:18:07.691 INFO:tasks.rados.rados.0.trial068.stdout:81: writing trial06842131-81 from 544560 to 1328010 tid 1 2026-02-25T10:18:07.695 INFO:tasks.rados.rados.0.trial068.stdout:81: writing trial06842131-81 from 1915030 to 2396378 tid 2 2026-02-25T10:18:07.699 INFO:tasks.rados.rados.0.trial068.stdout:81: writing trial06842131-81 from 2797604 to 3277365 tid 3 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout:75: finishing write tid 1 to trial06842131-75 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout:73: finishing write tid 1 to trial06842131-73 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout:48: finishing write tid 4 to trial06842131-48 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout:48: oid 48 updating version 0 to 15 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout:48: oid 48 version 15 is already newer than 13 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout:48: oid 48 version 15 is already newer than 15 2026-02-25T10:18:07.700 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2026-02-25T10:18:07.701 INFO:tasks.rados.rados.0.trial068.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2026-02-25T10:18:07.701 INFO:tasks.rados.rados.0.trial068.stdout:48: done (15 left) 2026-02-25T10:18:07.701 INFO:tasks.rados.rados.0.trial068.stdout:82: write initial oid 82 2026-02-25T10:18:07.701 INFO:tasks.rados.rados.0.trial068.stdout:82: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2026-02-25T10:18:07.702 INFO:tasks.rados.rados.0.trial068.stdout:82: writing trial06842131-82 from 561367 to 1020066 tid 1 2026-02-25T10:18:07.708 INFO:tasks.rados.rados.0.trial068.stdout:82: writing trial06842131-82 from 1573512 to 2314871 tid 2 2026-02-25T10:18:07.712 INFO:tasks.rados.rados.0.trial068.stdout:82: writing trial06842131-82 from 2941380 to 3361367 tid 3 2026-02-25T10:18:07.712 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.712 INFO:tasks.rados.rados.0.trial068.stdout:79: finishing write tid 1 to trial06842131-79 2026-02-25T10:18:07.712 INFO:tasks.rados.rados.0.trial068.stdout:75: finishing write tid 2 to trial06842131-75 2026-02-25T10:18:07.712 INFO:tasks.rados.rados.0.trial068.stdout:70: finishing write tid 1 to trial06842131-70 2026-02-25T10:18:07.712 INFO:tasks.rados.rados.0.trial068.stdout:73: finishing write tid 2 to trial06842131-73 2026-02-25T10:18:07.713 INFO:tasks.rados.rados.0.trial068.stdout:62: finishing write tid 1 to trial06842131-62 2026-02-25T10:18:07.722 INFO:tasks.rados.rados.0.trial068.stdout:74: finishing write tid 2 to trial06842131-74 2026-02-25T10:18:07.723 INFO:tasks.rados.rados.0.trial068.stdout:75: finishing write tid 3 to trial06842131-75 2026-02-25T10:18:07.727 INFO:tasks.rados.rados.0.trial068.stdout:56: finishing write tid 1 to trial06842131-56 2026-02-25T10:18:07.729 INFO:tasks.rados.rados.0.trial068.stdout:70: finishing write tid 2 to trial06842131-70 2026-02-25T10:18:07.737 INFO:tasks.rados.rados.0.trial068.stdout:62: finishing write tid 2 to trial06842131-62 2026-02-25T10:18:07.738 INFO:tasks.rados.rados.0.trial068.stdout:74: finishing write tid 3 to trial06842131-74 2026-02-25T10:18:07.739 INFO:tasks.rados.rados.0.trial068.stdout:75: finishing write tid 4 to trial06842131-75 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:75: finishing write tid 5 to trial06842131-75 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:75: oid 75 updating version 0 to 26 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:75: oid 75 version 26 is already newer than 26 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:75: oid 75 version 26 is already newer than 26 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:75: oid 75 version 26 is already newer than 23 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 75 v 26 (ObjNum 74 snap 0 seq_num 74) dirty exists 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:75: left oid 75 (ObjNum 74 snap 0 seq_num 74) 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:75: done (15 left) 2026-02-25T10:18:07.744 INFO:tasks.rados.rados.0.trial068.stdout:83: write initial oid 83 2026-02-25T10:18:07.745 INFO:tasks.rados.rados.0.trial068.stdout:83: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2026-02-25T10:18:07.747 INFO:tasks.rados.rados.0.trial068.stdout:83: writing trial06842131-83 from 578174 to 1112122 tid 1 2026-02-25T10:18:07.751 INFO:tasks.rados.rados.0.trial068.stdout:83: writing trial06842131-83 from 1515641 to 2000658 tid 2 2026-02-25T10:18:07.756 INFO:tasks.rados.rados.0.trial068.stdout:83: writing trial06842131-83 from 2736097 to 3238402 tid 3 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:81: finishing write tid 1 to trial06842131-81 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:74: finishing write tid 4 to trial06842131-74 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:74: finishing write tid 5 to trial06842131-74 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:74: oid 74 updating version 0 to 13 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:74: oid 74 updating version 13 to 16 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:74: oid 74 version 16 is already newer than 16 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:74: oid 74 version 16 is already newer than 16 2026-02-25T10:18:07.757 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 74 v 16 (ObjNum 73 snap 0 seq_num 73) dirty exists 2026-02-25T10:18:07.758 INFO:tasks.rados.rados.0.trial068.stdout:74: left oid 74 (ObjNum 73 snap 0 seq_num 73) 2026-02-25T10:18:07.758 INFO:tasks.rados.rados.0.trial068.stdout:74: done (15 left) 2026-02-25T10:18:07.758 INFO:tasks.rados.rados.0.trial068.stdout:84: write initial oid 84 2026-02-25T10:18:07.758 INFO:tasks.rados.rados.0.trial068.stdout:84: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2026-02-25T10:18:07.760 INFO:tasks.rados.rados.0.trial068.stdout:84: writing trial06842131-84 from 594981 to 1204178 tid 1 2026-02-25T10:18:07.766 INFO:tasks.rados.rados.0.trial068.stdout:84: writing trial06842131-84 from 1974123 to 2719151 tid 2 2026-02-25T10:18:07.769 INFO:tasks.rados.rados.0.trial068.stdout:84: writing trial06842131-84 from 3279873 to 3394981 tid 3 2026-02-25T10:18:07.769 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:62: finishing write tid 3 to trial06842131-62 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:81: finishing write tid 2 to trial06842131-81 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:62: finishing write tid 4 to trial06842131-62 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:62: oid 62 updating version 0 to 16 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:62: oid 62 version 16 is already newer than 14 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:62: oid 62 version 16 is already newer than 16 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 62 v 16 (ObjNum 61 snap 0 seq_num 61) dirty exists 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:62: left oid 62 (ObjNum 61 snap 0 seq_num 61) 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:73: finishing write tid 3 to trial06842131-73 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:62: done (15 left) 2026-02-25T10:18:07.770 INFO:tasks.rados.rados.0.trial068.stdout:85: write initial oid 85 2026-02-25T10:18:07.771 INFO:tasks.rados.rados.0.trial068.stdout:85: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2026-02-25T10:18:07.773 INFO:tasks.rados.rados.0.trial068.stdout:85: writing trial06842131-85 from 611788 to 1012587 tid 1 2026-02-25T10:18:07.777 INFO:tasks.rados.rados.0.trial068.stdout:85: writing trial06842131-85 from 1748958 to 2237644 tid 2 2026-02-25T10:18:07.782 INFO:tasks.rados.rados.0.trial068.stdout:85: writing trial06842131-85 from 2907296 to 3411788 tid 3 2026-02-25T10:18:07.783 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.783 INFO:tasks.rados.rados.0.trial068.stdout:70: finishing write tid 3 to trial06842131-70 2026-02-25T10:18:07.783 INFO:tasks.rados.rados.0.trial068.stdout:79: finishing write tid 2 to trial06842131-79 2026-02-25T10:18:07.783 INFO:tasks.rados.rados.0.trial068.stdout:81: finishing write tid 3 to trial06842131-81 2026-02-25T10:18:07.783 INFO:tasks.rados.rados.0.trial068.stdout:73: finishing write tid 4 to trial06842131-73 2026-02-25T10:18:07.784 INFO:tasks.rados.rados.0.trial068.stdout:83: finishing write tid 1 to trial06842131-83 2026-02-25T10:18:07.785 INFO:tasks.rados.rados.0.trial068.stdout:70: finishing write tid 4 to trial06842131-70 2026-02-25T10:18:07.785 INFO:tasks.rados.rados.0.trial068.stdout:81: finishing write tid 4 to trial06842131-81 2026-02-25T10:18:07.786 INFO:tasks.rados.rados.0.trial068.stdout:79: finishing write tid 3 to trial06842131-79 2026-02-25T10:18:07.787 INFO:tasks.rados.rados.0.trial068.stdout:81: finishing write tid 5 to trial06842131-81 2026-02-25T10:18:07.787 INFO:tasks.rados.rados.0.trial068.stdout:81: oid 81 updating version 0 to 19 2026-02-25T10:18:07.787 INFO:tasks.rados.rados.0.trial068.stdout:81: oid 81 version 19 is already newer than 19 2026-02-25T10:18:07.787 INFO:tasks.rados.rados.0.trial068.stdout:81: oid 81 version 19 is already newer than 16 2026-02-25T10:18:07.788 INFO:tasks.rados.rados.0.trial068.stdout:81: oid 81 version 19 is already newer than 19 2026-02-25T10:18:07.788 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 81 v 19 (ObjNum 80 snap 0 seq_num 80) dirty exists 2026-02-25T10:18:07.788 INFO:tasks.rados.rados.0.trial068.stdout:81: left oid 81 (ObjNum 80 snap 0 seq_num 80) 2026-02-25T10:18:07.788 INFO:tasks.rados.rados.0.trial068.stdout:81: done (15 left) 2026-02-25T10:18:07.788 INFO:tasks.rados.rados.0.trial068.stdout:86: write initial oid 86 2026-02-25T10:18:07.788 INFO:tasks.rados.rados.0.trial068.stdout:86: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2026-02-25T10:18:07.791 INFO:tasks.rados.rados.0.trial068.stdout:86: writing trial06842131-86 from 628595 to 1104643 tid 1 2026-02-25T10:18:07.797 INFO:tasks.rados.rados.0.trial068.stdout:86: writing trial06842131-86 from 1807440 to 2439784 tid 2 2026-02-25T10:18:07.801 INFO:tasks.rados.rados.0.trial068.stdout:86: writing trial06842131-86 from 2934719 to 3387193 tid 3 2026-02-25T10:18:07.801 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:70: finishing write tid 5 to trial06842131-70 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:70: oid 70 updating version 0 to 20 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:70: oid 70 version 20 is already newer than 20 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:70: oid 70 version 20 is already newer than 17 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:70: oid 70 version 20 is already newer than 20 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 70 v 20 (ObjNum 69 snap 0 seq_num 69) dirty exists 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:70: left oid 70 (ObjNum 69 snap 0 seq_num 69) 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:73: finishing write tid 5 to trial06842131-73 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:73: oid 73 updating version 0 to 25 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:73: oid 73 version 25 is already newer than 25 2026-02-25T10:18:07.802 INFO:tasks.rados.rados.0.trial068.stdout:73: oid 73 version 25 is already newer than 22 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:73: oid 73 version 25 is already newer than 25 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 73 v 25 (ObjNum 72 snap 0 seq_num 72) dirty exists 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:73: left oid 73 (ObjNum 72 snap 0 seq_num 72) 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:79: finishing write tid 4 to trial06842131-79 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:70: done (15 left) 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:73: done (14 left) 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:87: write initial oid 87 2026-02-25T10:18:07.803 INFO:tasks.rados.rados.0.trial068.stdout:87: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2026-02-25T10:18:07.805 INFO:tasks.rados.rados.0.trial068.stdout:87: writing trial06842131-87 from 645402 to 1196699 tid 1 2026-02-25T10:18:07.809 INFO:tasks.rados.rados.0.trial068.stdout:87: writing trial06842131-87 from 1749569 to 2241924 tid 2 2026-02-25T10:18:07.814 INFO:tasks.rados.rados.0.trial068.stdout:87: writing trial06842131-87 from 2845789 to 3445402 tid 3 2026-02-25T10:18:07.814 INFO:tasks.rados.rados.0.trial068.stdout:88: write initial oid 88 2026-02-25T10:18:07.814 INFO:tasks.rados.rados.0.trial068.stdout:88: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2026-02-25T10:18:07.818 INFO:tasks.rados.rados.0.trial068.stdout:88: writing trial06842131-88 from 662209 to 1288755 tid 1 2026-02-25T10:18:07.823 INFO:tasks.rados.rados.0.trial068.stdout:88: writing trial06842131-88 from 1808051 to 2444064 tid 2 2026-02-25T10:18:07.827 INFO:tasks.rados.rados.0.trial068.stdout:88: writing trial06842131-88 from 2873212 to 3462209 tid 3 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:56: finishing write tid 2 to trial06842131-56 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:79: finishing write tid 5 to trial06842131-79 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:79: oid 79 updating version 0 to 10 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:79: oid 79 version 10 is already newer than 10 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:79: oid 79 version 10 is already newer than 7 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:79: oid 79 version 10 is already newer than 10 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 79 v 10 (ObjNum 78 snap 0 seq_num 78) dirty exists 2026-02-25T10:18:07.828 INFO:tasks.rados.rados.0.trial068.stdout:79: left oid 79 (ObjNum 78 snap 0 seq_num 78) 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:83: finishing write tid 2 to trial06842131-83 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:78: finishing write tid 1 to trial06842131-78 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:56: finishing write tid 3 to trial06842131-56 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:80: finishing write tid 1 to trial06842131-80 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:83: finishing write tid 3 to trial06842131-83 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:56: finishing write tid 4 to trial06842131-56 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:83: finishing write tid 4 to trial06842131-83 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:79: done (15 left) 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:89: write initial oid 89 2026-02-25T10:18:07.829 INFO:tasks.rados.rados.0.trial068.stdout:89: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2026-02-25T10:18:07.832 INFO:tasks.rados.rados.0.trial068.stdout:89: writing trial06842131-89 from 679016 to 1380811 tid 1 2026-02-25T10:18:07.836 INFO:tasks.rados.rados.0.trial068.stdout:89: writing trial06842131-89 from 1866533 to 2362557 tid 2 2026-02-25T10:18:07.840 INFO:tasks.rados.rados.0.trial068.stdout:89: writing trial06842131-89 from 2900635 to 3303278 tid 3 2026-02-25T10:18:07.840 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:83: finishing write tid 5 to trial06842131-83 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:83: oid 83 updating version 0 to 16 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:83: oid 83 version 16 is already newer than 16 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:83: oid 83 version 16 is already newer than 16 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:83: oid 83 version 16 is already newer than 13 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 83 v 16 (ObjNum 82 snap 0 seq_num 82) dirty exists 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:83: left oid 83 (ObjNum 82 snap 0 seq_num 82) 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:78: finishing write tid 2 to trial06842131-78 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:56: finishing write tid 5 to trial06842131-56 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:56: oid 56 updating version 0 to 19 2026-02-25T10:18:07.841 INFO:tasks.rados.rados.0.trial068.stdout:56: oid 56 version 19 is already newer than 19 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:56: oid 56 version 19 is already newer than 16 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:56: oid 56 version 19 is already newer than 19 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 56 v 19 (ObjNum 55 snap 0 seq_num 55) dirty exists 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:56: left oid 56 (ObjNum 55 snap 0 seq_num 55) 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:86: finishing write tid 1 to trial06842131-86 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:56: done (15 left) 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:83: done (14 left) 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:90: write initial oid 90 2026-02-25T10:18:07.842 INFO:tasks.rados.rados.0.trial068.stdout:90: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2026-02-25T10:18:07.845 INFO:tasks.rados.rados.0.trial068.stdout:90: writing trial06842131-90 from 695823 to 1472867 tid 1 2026-02-25T10:18:07.849 INFO:tasks.rados.rados.0.trial068.stdout:90: writing trial06842131-90 from 1925015 to 2564697 tid 2 2026-02-25T10:18:07.853 INFO:tasks.rados.rados.0.trial068.stdout:90: writing trial06842131-90 from 3328058 to 3495823 tid 3 2026-02-25T10:18:07.853 INFO:tasks.rados.rados.0.trial068.stdout:91: write initial oid 91 2026-02-25T10:18:07.853 INFO:tasks.rados.rados.0.trial068.stdout:91: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2026-02-25T10:18:07.857 INFO:tasks.rados.rados.0.trial068.stdout:91: writing trial06842131-91 from 712630 to 1164923 tid 1 2026-02-25T10:18:07.861 INFO:tasks.rados.rados.0.trial068.stdout:91: writing trial06842131-91 from 1583497 to 2083190 tid 2 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout:91: writing trial06842131-91 from 2555481 to 2980668 tid 3 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout:80: finishing write tid 2 to trial06842131-80 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout:64: finishing write tid 1 to trial06842131-64 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout:84: finishing write tid 1 to trial06842131-84 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout:61: finishing write tid 1 to trial06842131-61 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout:86: finishing write tid 2 to trial06842131-86 2026-02-25T10:18:07.865 INFO:tasks.rados.rados.0.trial068.stdout:78: finishing write tid 3 to trial06842131-78 2026-02-25T10:18:07.871 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T10:18:07.871 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T10:18:07.871 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3] 2026-02-25T10:18:07.872 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-25T10:18:07.872 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T10:18:07.875 INFO:tasks.rados.rados.0.trial068.stdout:84: finishing write tid 2 to trial06842131-84 2026-02-25T10:18:07.960 INFO:tasks.ceph.osd.3:Stopped 2026-02-25T10:18:09.187 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:18:09.265 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:09.262+0000 7fdd7dfed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:09.267 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:09.266+0000 7fdd7dfed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:12.960 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2026-02-25T10:18:12.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T10:18:12.961 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2] 2026-02-25T10:18:12.961 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T10:18:12.961 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T10:18:13.037 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T10:18:18.038 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 2] live_osds: [0, 1] 2026-02-25T10:18:18.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T10:18:18.039 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-25T10:18:18.039 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T10:18:18.039 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T10:18:18.042 INFO:tasks.ceph.osd.2:Started 2026-02-25T10:18:18.042 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:18.113 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:18.116 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:22.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-25T10:18:23.117 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:23.189 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:23.192 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:27.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2026-02-25T10:18:28.192 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:28.265 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:28.269 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:32.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2026-02-25T10:18:33.269 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:33.341 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:33.345 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:37.277 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:37.281 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:38.311 INFO:tasks.rados.rados.0.trial068.stdout:80: finishing write tid 3 to trial06842131-80 2026-02-25T10:18:38.312 INFO:tasks.rados.rados.0.trial068.stdout:80: finishing write tid 4 to trial06842131-80 2026-02-25T10:18:38.313 INFO:tasks.rados.rados.0.trial068.stdout:82: finishing write tid 1 to trial06842131-82 2026-02-25T10:18:38.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2026-02-25T10:18:38.334 INFO:tasks.rados.rados.0.trial068.stdout:80: finishing write tid 5 to trial06842131-80 2026-02-25T10:18:38.334 INFO:tasks.rados.rados.0.trial068.stdout:80: oid 80 updating version 0 to 29 2026-02-25T10:18:38.334 INFO:tasks.rados.rados.0.trial068.stdout:80: oid 80 version 29 is already newer than 26 2026-02-25T10:18:38.334 INFO:tasks.rados.rados.0.trial068.stdout:80: oid 80 version 29 is already newer than 29 2026-02-25T10:18:38.334 INFO:tasks.rados.rados.0.trial068.stdout:80: oid 80 version 29 is already newer than 28 2026-02-25T10:18:38.334 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 80 v 29 (ObjNum 79 snap 0 seq_num 79) dirty exists 2026-02-25T10:18:38.335 INFO:tasks.rados.rados.0.trial068.stdout:80: left oid 80 (ObjNum 79 snap 0 seq_num 79) 2026-02-25T10:18:38.335 INFO:tasks.rados.rados.0.trial068.stdout:80: done (15 left) 2026-02-25T10:18:38.335 INFO:tasks.rados.rados.0.trial068.stdout:92: write initial oid 92 2026-02-25T10:18:38.335 INFO:tasks.rados.rados.0.trial068.stdout:92: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2026-02-25T10:18:38.340 INFO:tasks.rados.rados.0.trial068.stdout:92: writing trial06842131-92 from 729437 to 1256979 tid 1 2026-02-25T10:18:38.345 INFO:tasks.rados.rados.0.trial068.stdout:92: writing trial06842131-92 from 1925626 to 2568977 tid 2 2026-02-25T10:18:38.346 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:38.350 INFO:tasks.rados.rados.0.trial068.stdout:92: writing trial06842131-92 from 3266551 to 3529437 tid 3 2026-02-25T10:18:38.350 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:38.350 INFO:tasks.rados.rados.0.trial068.stdout:82: finishing write tid 2 to trial06842131-82 2026-02-25T10:18:38.355 INFO:tasks.rados.rados.0.trial068.stdout:82: finishing write tid 3 to trial06842131-82 2026-02-25T10:18:38.361 INFO:tasks.rados.rados.0.trial068.stdout:82: finishing write tid 4 to trial06842131-82 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:82: finishing write tid 5 to trial06842131-82 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:82: oid 82 updating version 0 to 33 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:82: oid 82 version 33 is already newer than 30 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:82: oid 82 version 33 is already newer than 33 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:82: oid 82 version 33 is already newer than 33 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 82 v 33 (ObjNum 81 snap 0 seq_num 81) dirty exists 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:82: left oid 82 (ObjNum 81 snap 0 seq_num 81) 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:82: done (15 left) 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:93: write initial oid 93 2026-02-25T10:18:38.366 INFO:tasks.rados.rados.0.trial068.stdout:93: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2026-02-25T10:18:38.372 INFO:tasks.rados.rados.0.trial068.stdout:93: writing trial06842131-93 from 746244 to 1465388 tid 1 2026-02-25T10:18:38.377 INFO:tasks.rados.rados.0.trial068.stdout:93: writing trial06842131-93 from 2100461 to 2603823 tid 2 2026-02-25T10:18:38.381 INFO:tasks.rados.rados.0.trial068.stdout:93: writing trial06842131-93 from 3126680 to 3546244 tid 3 2026-02-25T10:18:38.381 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:38.381 INFO:tasks.rados.rados.0.trial068.stdout:90: finishing write tid 1 to trial06842131-90 2026-02-25T10:18:38.386 INFO:tasks.rados.rados.0.trial068.stdout:90: finishing write tid 2 to trial06842131-90 2026-02-25T10:18:38.391 INFO:tasks.rados.rados.0.trial068.stdout:93: finishing write tid 1 to trial06842131-93 2026-02-25T10:18:38.392 INFO:tasks.rados.rados.0.trial068.stdout:90: finishing write tid 3 to trial06842131-90 2026-02-25T10:18:38.399 INFO:tasks.rados.rados.0.trial068.stdout:90: finishing write tid 4 to trial06842131-90 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:90: finishing write tid 5 to trial06842131-90 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:90: oid 90 updating version 0 to 37 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:90: oid 90 version 37 is already newer than 34 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:90: oid 90 version 37 is already newer than 37 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:90: oid 90 version 37 is already newer than 37 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 90 v 37 (ObjNum 89 snap 0 seq_num 89) dirty exists 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:90: left oid 90 (ObjNum 89 snap 0 seq_num 89) 2026-02-25T10:18:38.402 INFO:tasks.rados.rados.0.trial068.stdout:90: done (15 left) 2026-02-25T10:18:38.403 INFO:tasks.rados.rados.0.trial068.stdout:94: write initial oid 94 2026-02-25T10:18:38.403 INFO:tasks.rados.rados.0.trial068.stdout:94: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2026-02-25T10:18:38.407 INFO:tasks.rados.rados.0.trial068.stdout:94: writing trial06842131-94 from 763051 to 1557444 tid 1 2026-02-25T10:18:38.412 INFO:tasks.rados.rados.0.trial068.stdout:94: writing trial06842131-94 from 2158943 to 2805963 tid 2 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout:94: writing trial06842131-94 from 3437750 to 3563051 tid 3 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout:93: finishing write tid 2 to trial06842131-93 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout:93: finishing write tid 3 to trial06842131-93 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout:93: finishing write tid 4 to trial06842131-93 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout:93: finishing write tid 5 to trial06842131-93 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout:93: oid 93 updating version 0 to 16 2026-02-25T10:18:38.416 INFO:tasks.rados.rados.0.trial068.stdout:93: oid 93 version 16 is already newer than 13 2026-02-25T10:18:38.417 INFO:tasks.rados.rados.0.trial068.stdout:93: oid 93 version 16 is already newer than 16 2026-02-25T10:18:38.417 INFO:tasks.rados.rados.0.trial068.stdout:93: oid 93 version 16 is already newer than 16 2026-02-25T10:18:38.417 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 93 v 16 (ObjNum 92 snap 0 seq_num 92) dirty exists 2026-02-25T10:18:38.417 INFO:tasks.rados.rados.0.trial068.stdout:93: left oid 93 (ObjNum 92 snap 0 seq_num 92) 2026-02-25T10:18:38.417 INFO:tasks.rados.rados.0.trial068.stdout:93: done (15 left) 2026-02-25T10:18:38.417 INFO:tasks.rados.rados.0.trial068.stdout:95: write initial oid 95 2026-02-25T10:18:38.417 INFO:tasks.rados.rados.0.trial068.stdout:95: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2026-02-25T10:18:38.419 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:38.420 INFO:tasks.rados.rados.0.trial068.stdout:95: writing trial06842131-95 from 779858 to 1249500 tid 1 2026-02-25T10:18:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:38.422 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:38.424 INFO:tasks.rados.rados.0.trial068.stdout:95: writing trial06842131-95 from 1817425 to 2324456 tid 2 2026-02-25T10:18:38.428 INFO:tasks.rados.rados.0.trial068.stdout:95: writing trial06842131-95 from 2781526 to 3368154 tid 3 2026-02-25T10:18:38.428 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:39.282 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:18:39.358 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:39.356+0000 7f46d0fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:39.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:39.359+0000 7f46d0fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:43.423 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:43.495 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:43.499 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:43.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2026-02-25T10:18:44.366 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:44.367 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:44.373 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:44.398 INFO:tasks.rados.rados.0.trial068.stdout:92: finishing write tid 1 to trial06842131-92 2026-02-25T10:18:44.423 INFO:tasks.rados.rados.0.trial068.stdout:92: finishing write tid 2 to trial06842131-92 2026-02-25T10:18:44.426 INFO:tasks.rados.rados.0.trial068.stdout:92: finishing write tid 3 to trial06842131-92 2026-02-25T10:18:44.431 INFO:tasks.rados.rados.0.trial068.stdout:92: finishing write tid 4 to trial06842131-92 2026-02-25T10:18:44.433 INFO:tasks.rados.rados.0.trial068.stdout:92: finishing write tid 5 to trial06842131-92 2026-02-25T10:18:44.433 INFO:tasks.rados.rados.0.trial068.stdout:92: oid 92 updating version 0 to 14 2026-02-25T10:18:44.433 INFO:tasks.rados.rados.0.trial068.stdout:92: oid 92 version 14 is already newer than 14 2026-02-25T10:18:44.433 INFO:tasks.rados.rados.0.trial068.stdout:92: oid 92 version 14 is already newer than 14 2026-02-25T10:18:44.433 INFO:tasks.rados.rados.0.trial068.stdout:92: oid 92 version 14 is already newer than 11 2026-02-25T10:18:44.434 INFO:tasks.rados.rados.0.trial068.stdout:update_object_version oid 92 v 14 (ObjNum 91 snap 0 seq_num 91) dirty exists 2026-02-25T10:18:44.434 INFO:tasks.rados.rados.0.trial068.stdout:92: left oid 92 (ObjNum 91 snap 0 seq_num 91) 2026-02-25T10:18:44.434 INFO:tasks.rados.rados.0.trial068.stdout:92: done (15 left) 2026-02-25T10:18:44.434 INFO:tasks.rados.rados.0.trial068.stdout:96: write initial oid 96 2026-02-25T10:18:44.434 INFO:tasks.rados.rados.0.trial068.stdout:96: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2026-02-25T10:18:44.438 INFO:tasks.rados.rados.0.trial068.stdout:96: writing trial06842131-96 from 796665 to 1341556 tid 1 2026-02-25T10:18:44.442 INFO:tasks.rados.rados.0.trial068.stdout:96: writing trial06842131-96 from 1759554 to 2410243 tid 2 2026-02-25T10:18:44.446 INFO:tasks.rados.rados.0.trial068.stdout:96: writing trial06842131-96 from 2976243 to 3596665 tid 3 2026-02-25T10:18:44.447 INFO:tasks.rados.rados.0.trial068.stdout: waiting on 16 2026-02-25T10:18:46.374 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:18:46.453 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:46.450+0000 7fc4406cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:46.455 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:46.454+0000 7fc4406cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:46.481 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:46.481 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:46.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:46.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:48.485 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:18:48.500 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:48.565 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:48.562+0000 7f13efb5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:48.567 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:48.566+0000 7f13efb5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:48.571 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:48.574 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:48.596 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:48.596 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:48.598 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:49.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~27s 2026-02-25T10:18:50.599 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:18:50.677 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:50.674+0000 7f423ee5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:50.679 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:50.678+0000 7f423ee5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:50.713 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:50.713 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:50.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:52.718 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:18:52.796 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:52.793+0000 7f5f54ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:52.798 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:52.796+0000 7f5f54ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:52.826 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:52.826 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:52.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:53.576 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:53.649 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:53.652 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:54.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~32s 2026-02-25T10:18:54.831 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:18:54.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:54.907+0000 7f6ddcecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:54.913 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:54.911+0000 7f6ddcecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:54.939 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:54.939 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:54.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:56.943 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:18:57.021 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:57.018+0000 7f56a48c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:57.023 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:57.022+0000 7f56a48c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:57.050 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:57.050 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:57.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:58.653 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:18:58.725 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:18:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:18:58.728 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:18:59.054 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:18:59.133 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:59.130+0000 7f08fd30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:59.136 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:18:59.134+0000 7f08fd30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:18:59.162 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:18:59.162 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:18:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:18:59.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:18:59.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~38s 2026-02-25T10:19:01.168 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:01.246 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:01.243+0000 7f8185774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:01.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:01.247+0000 7f8185774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:01.275 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:01.275 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:01.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:03.280 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:03.359 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:03.356+0000 7f48cba8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:03.361 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:03.359+0000 7f48cba8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:03.389 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:03.389 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:03.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:03.729 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:03.801 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:03.804 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:05.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~43s 2026-02-25T10:19:05.392 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:05.472 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:05.469+0000 7f8c0aa1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:05.474 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:05.473+0000 7f8c0aa1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:05.502 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:05.502 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:05.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:07.506 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:07.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:07.583+0000 7fd9e865a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:07.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:07.586+0000 7fd9e865a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:07.615 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:07.616 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:07.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:08.805 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:08.876 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:08.880 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:09.619 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:09.695 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:09.692+0000 7f9daa6b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:09.698 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:09.696+0000 7f9daa6b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:09.726 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:09.727 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:09.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:10.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~49s 2026-02-25T10:19:11.732 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:11.808 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:11.805+0000 7f9f64d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:11.810 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:11.808+0000 7f9f64d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:11.837 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:11.837 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:11.839 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:13.841 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:13.880 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:13.919 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:13.916+0000 7f9ddbe46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:13.922 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:13.920+0000 7f9ddbe46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:13.949 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:13.949 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:13.950 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:13.952 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:13.953 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:15.953 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:16.032 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:16.029+0000 7f148aacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:16.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:16.032+0000 7f148aacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:16.062 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:16.062 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:16.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:16.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~54s 2026-02-25T10:19:18.067 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:18.144 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:18.141+0000 7fac2865a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:18.146 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:18.144+0000 7fac2865a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:18.173 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:18.173 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:18.178 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:18.955 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:19.027 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:19.029 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:20.179 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:20.257 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:20.255+0000 7f82079a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:20.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:20.258+0000 7f82079a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:20.286 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:20.286 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:20.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:21.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~59s 2026-02-25T10:19:22.289 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:22.367 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:22.364+0000 7f87c161a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:22.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:22.368+0000 7f87c161a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:22.395 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:22.396 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:22.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:24.031 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:24.103 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:24.106 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:24.399 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:24.475 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:24.471+0000 7fa288f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:24.477 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:24.475+0000 7fa288f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:24.503 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:24.504 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:24.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:26.507 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:26.586 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:26.582+0000 7ff38470b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:26.588 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:26.586+0000 7ff38470b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:26.617 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:26.617 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:26.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:26.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s 2026-02-25T10:19:28.621 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:28.699 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:28.696+0000 7fe5b2b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:28.702 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:28.700+0000 7fe5b2b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:28.727 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:28.728 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:28.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:29.108 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:29.177 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:29.181 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:30.730 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:30.809 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:30.806+0000 7f9793046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:30.811 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:30.809+0000 7f9793046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:30.838 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:30.838 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:30.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:32.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~70s 2026-02-25T10:19:32.840 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:32.918 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:32.915+0000 7fc2565b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:32.920 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:32.919+0000 7fc2565b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:32.947 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:32.947 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:32.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:34.181 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:34.252 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:34.255 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:34.950 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:35.028 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:35.025+0000 7fbeb1dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:35.031 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:35.029+0000 7fbeb1dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:35.057 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:35.058 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:35.060 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:37.061 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:37.139 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:37.137+0000 7f6ea9e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:37.141 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:37.140+0000 7f6ea9e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:37.168 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:37.168 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:37.169 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:37.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~76s 2026-02-25T10:19:39.170 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:39.248 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:39.245+0000 7fca70a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:39.251 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:39.249+0000 7fca70a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:39.256 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:39.279 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:39.279 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:39.283 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:39.328 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:39.331 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:41.283 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:41.362 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:41.359+0000 7f1da83c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:41.364 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:41.363+0000 7f1da83c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:41.397 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:41.397 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:41.401 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:43.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~81s 2026-02-25T10:19:43.402 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:43.480 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:43.477+0000 7fa63b90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:43.483 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:43.482+0000 7fa63b90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:43.510 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:43.510 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:43.512 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:44.332 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:44.402 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:44.405 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:45.513 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:45.593 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:45.590+0000 7fe48fb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:45.595 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:45.593+0000 7fe48fb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:45.621 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:45.622 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:45.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:47.626 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:47.705 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:47.702+0000 7f81b39f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:47.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:47.706+0000 7f81b39f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:47.734 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:47.734 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:47.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:48.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~86s 2026-02-25T10:19:49.407 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:49.478 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:49.481 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:49.738 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:49.813 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:49.810+0000 7fb9fe25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:49.816 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:49.814+0000 7fb9fe25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:49.843 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:49.843 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:49.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:51.846 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:51.924 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:51.921+0000 7ff9ad6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:51.927 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:51.925+0000 7ff9ad6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:51.954 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:51.954 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:51.956 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:53.956 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:53.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~92s 2026-02-25T10:19:54.034 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:54.031+0000 7fdb5f85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:54.037 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:54.036+0000 7fdb5f85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:54.064 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:54.064 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:54.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:54.482 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:54.555 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:54.558 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:19:56.068 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:19:56.147 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:56.144+0000 7f0fa490b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:56.149 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:56.147+0000 7f0fa490b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:56.176 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:56.177 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:56.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:58.182 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:19:58.260 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:58.257+0000 7f61bfd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:58.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:19:58.260+0000 7f61bfd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:19:58.288 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:19:58.289 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:19:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:19:58.291 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:19:59.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~97s 2026-02-25T10:19:59.559 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:19:59.630 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:19:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:19:59.633 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:00.292 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:00.369 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:00.366+0000 7fab959f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:00.372 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:00.370+0000 7fab959f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:00.400 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:00.400 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:00.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:02.405 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:02.484 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:02.481+0000 7f4f4bf7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:02.486 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:02.485+0000 7f4f4bf7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:02.513 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:02.513 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:02.515 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:04.516 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:04.594 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:04.591+0000 7f8cd530b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:04.596 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:04.595+0000 7f8cd530b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:04.624 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:04.624 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:04.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:04.635 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:04.706 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:04.709 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:04.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~103s 2026-02-25T10:20:06.627 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:06.705 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:06.702+0000 7f11ae7ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:06.707 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:06.705+0000 7f11ae7ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:06.733 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:06.734 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:06.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:08.737 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:08.817 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:08.814+0000 7f07fd50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:08.819 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:08.818+0000 7f07fd50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:08.846 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:08.846 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:08.847 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:08.848 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:09.711 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:09.783 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:09.787 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:10.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~108s 2026-02-25T10:20:10.849 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:10.925 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:10.922+0000 7f331790b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:10.927 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:10.926+0000 7f331790b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:10.955 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:10.956 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:10.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:12.961 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:13.038 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:13.035+0000 7fcca47dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:13.040 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:13.039+0000 7fcca47dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:13.067 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:13.067 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:13.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:14.788 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:14.861 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:14.864 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:15.072 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:15.151 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:15.148+0000 7f9df52cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:15.153 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:15.151+0000 7f9df52cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:15.180 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:15.181 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:15.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:15.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~113s 2026-02-25T10:20:17.184 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:17.262 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:17.259+0000 7fb574846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:17.264 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:17.263+0000 7fb574846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:17.291 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:17.291 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:17.293 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:19.294 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:19.372 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:19.369+0000 7f85496b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:19.374 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:19.372+0000 7f85496b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:19.401 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:19.401 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:19.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:19.866 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:19.937 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:19.940 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:20.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~119s 2026-02-25T10:20:21.407 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:21.487 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:21.484+0000 7fab17acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:21.490 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:21.488+0000 7fab17acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:21.517 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:21.517 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:21.521 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:23.522 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:23.601 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:23.598+0000 7f1dda10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:23.603 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:23.602+0000 7f1dda10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:23.635 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:23.636 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:23.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:24.941 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:25.012 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:25.015 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:25.640 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:25.717 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:25.714+0000 7f2e8525a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:25.719 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:25.717+0000 7f2e8525a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:25.745 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:25.745 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:25.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:26.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~124s 2026-02-25T10:20:27.750 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:27.828 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:27.824+0000 7f9ab265a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:27.830 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:27.828+0000 7f9ab265a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:27.857 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:27.857 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:27.859 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:29.860 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:29.939 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:29.936+0000 7fc423dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:29.941 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:29.939+0000 7fc423dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:29.975 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:29.975 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:29.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:30.017 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:30.088 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:30.091 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:31.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2026-02-25T10:20:31.980 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:32.059 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:32.056+0000 7f76482cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:32.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:32.060+0000 7f76482cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:32.090 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:32.090 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:32.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:34.096 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:34.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:34.172+0000 7f1dfb25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:34.177 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:34.176+0000 7f1dfb25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:34.203 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:34.203 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:34.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:35.092 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:35.163 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:35.167 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:36.208 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:36.287 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:36.284+0000 7f6c9cd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:36.290 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:36.288+0000 7f6c9cd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:36.318 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:36.318 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:36.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:37.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2026-02-25T10:20:38.320 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:38.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:38.394+0000 7f98a7b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:38.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:38.398+0000 7f98a7b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:38.428 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:38.428 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:38.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:40.168 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:40.239 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:40.242 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:40.433 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:40.511 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:40.507+0000 7fa8fcbc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:40.513 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:40.511+0000 7fa8fcbc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:40.541 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:40.541 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:40.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:42.545 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:42.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2026-02-25T10:20:42.622 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:42.619+0000 7f63b6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:42.624 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:42.623+0000 7f63b6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:42.651 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:42.651 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:42.653 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:44.653 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:44.732 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:44.729+0000 7fb7ce05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:44.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:44.732+0000 7fb7ce05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:44.761 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:44.761 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:44.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:45.243 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:45.320 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:45.325 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:46.765 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:46.842 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:46.839+0000 7f835730b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:46.844 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:46.843+0000 7f835730b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:46.871 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:46.871 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:46.873 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:48.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~146s 2026-02-25T10:20:48.874 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:48.951 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:48.948+0000 7f8c9290b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:48.954 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:48.952+0000 7f8c9290b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:48.980 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:48.980 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:48.982 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:50.326 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:50.397 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:50.400 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:50.983 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:51.062 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:51.060+0000 7f44e9f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:51.065 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:51.065+0000 7f44e9f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:51.092 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:51.092 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:51.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:53.095 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:53.173 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:53.170+0000 7f7a4405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:53.175 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:53.174+0000 7f7a4405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:53.202 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:53.203 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:53.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:53.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~151s 2026-02-25T10:20:55.205 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:55.285 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:55.283+0000 7f18dca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:55.287 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:55.286+0000 7f18dca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:55.314 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:55.315 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:55.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:55.401 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:20:55.473 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:20:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:20:55.477 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:20:57.320 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:20:57.398 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:57.395+0000 7f3b57562640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:57.400 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:57.400+0000 7f3b57562640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:57.426 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:57.427 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:57.431 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:20:58.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~157s 2026-02-25T10:20:59.432 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:20:59.510 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:59.508+0000 7f576099c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:59.512 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:20:59.512+0000 7f576099c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:20:59.539 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:20:59.539 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:20:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:20:59.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:00.478 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:00.549 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:00.553 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:01.544 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:01.622 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:01.620+0000 7f0409c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:01.625 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:01.624+0000 7f0409c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:01.651 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:01.651 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:01.653 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:03.655 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:03.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:03.731+0000 7fba2ab5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:03.736 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:03.735+0000 7fba2ab5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:03.762 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:03.763 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:03.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:04.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~162s 2026-02-25T10:21:05.554 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:05.626 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:05.630 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:05.768 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:05.847 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:05.845+0000 7f9bb55dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:05.849 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:05.848+0000 7f9bb55dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:05.875 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:05.875 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:05.877 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:05.877 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:07.878 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:07.957 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:07.955+0000 7f8ced75c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:07.959 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:07.959+0000 7f8ced75c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:07.987 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:07.987 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:07.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:09.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~168s 2026-02-25T10:21:09.992 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:10.070 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:10.067+0000 7fbfee25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:10.072 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:10.071+0000 7fbfee25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:10.100 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:10.100 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:10.104 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:10.631 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:10.702 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:10.705 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:12.105 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:12.183 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:12.181+0000 7f561ee5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:12.186 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:12.185+0000 7f561ee5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:12.212 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:12.212 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:12.214 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:14.216 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:14.294 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:14.292+0000 7f4a1181a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:14.296 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:14.295+0000 7f4a1181a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:14.324 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:14.324 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:14.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:15.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~173s 2026-02-25T10:21:15.706 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:15.778 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:15.781 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:16.329 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:16.406 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:16.403+0000 7f6d5d8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:16.408 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:16.407+0000 7f6d5d8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:16.435 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:16.436 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:16.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:18.439 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:18.516 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:18.514+0000 7fb844846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:18.518 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:18.518+0000 7fb844846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:18.545 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:18.545 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:18.547 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:20.433 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~178s 2026-02-25T10:21:20.548 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:20.626 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:20.624+0000 7fbba7b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:20.629 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:20.628+0000 7fbba7b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:20.655 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:20.655 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:20.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:20.782 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:20.854 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:20.857 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:22.658 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:22.737 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:22.735+0000 7fdb5f6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:22.739 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:22.739+0000 7fdb5f6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:22.767 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:22.767 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:22.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:24.771 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:24.848 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:24.845+0000 7f5c0d5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:24.850 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:24.849+0000 7f5c0d5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:24.877 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:24.877 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:24.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:25.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~184s 2026-02-25T10:21:25.858 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:25.932 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:25.935 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:26.881 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:26.961 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:26.959+0000 7f721cccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:26.963 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:26.962+0000 7f721cccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:26.990 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:26.990 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:26.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:28.994 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:29.075 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:29.072+0000 7f559c051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:29.077 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:29.076+0000 7f559c051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:29.104 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:29.104 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:29.106 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:30.936 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:31.009 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:31.012 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:31.108 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:31.186 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:31.184+0000 7f9b62f6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:31.188 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:31.188+0000 7f9b62f6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:31.217 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:31.217 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:31.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:31.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~189s 2026-02-25T10:21:33.221 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:33.299 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:33.296+0000 7f6b5e846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:33.301 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:33.300+0000 7f6b5e846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:33.327 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:33.327 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:33.330 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:33.330 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:35.331 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:35.411 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:35.409+0000 7f36e3e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:35.413 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:35.412+0000 7f36e3e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:35.439 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:35.439 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:35.443 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:35.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:36.014 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:36.086 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:36.090 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:36.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~195s 2026-02-25T10:21:37.445 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:37.520 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:37.518+0000 7f0275bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:37.523 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:37.522+0000 7f027492f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:37.550 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:37.550 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:37.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:39.555 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:39.633 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:39.631+0000 7f6afe811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:39.636 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:39.635+0000 7f6afe811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:39.663 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:39.663 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:39.668 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:41.090 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:41.161 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:41.164 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:41.669 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:41.747 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:41.745+0000 7f550aecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:41.750 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:41.749+0000 7f550aecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:41.777 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:41.777 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:41.779 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:42.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~200s 2026-02-25T10:21:43.779 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:43.857 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:43.856+0000 7f9577e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:43.859 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:43.859+0000 7f9577e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:43.886 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:43.886 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:43.890 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:45.891 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:45.970 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:45.968+0000 7f8a81606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:45.972 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:45.971+0000 7f8a81606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:46.000 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:46.000 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:46.004 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:46.166 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:46.238 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:46.241 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:47.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2026-02-25T10:21:48.006 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:48.083 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:48.080+0000 7fdb05acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:48.085 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:48.084+0000 7fdb05acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:48.119 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:48.119 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:48.123 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:50.124 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:50.200 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:50.198+0000 7f2ce795b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:50.202 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:50.202+0000 7f2ce795b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:50.230 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:50.230 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:50.234 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:51.242 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:51.312 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:51.315 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:52.235 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:52.314 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:52.312+0000 7f01ab88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:52.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:52.315+0000 7f01ab88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:52.343 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:52.343 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:52.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:52.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~211s 2026-02-25T10:21:54.347 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:54.426 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:54.424+0000 7f8f7ec1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:54.428 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:54.427+0000 7f8f7ec1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:54.455 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:54.455 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:54.457 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:56.315 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:21:56.387 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:21:56.389 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:21:56.390 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:21:56.458 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:21:56.540 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:56.538+0000 7f1a4f23d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:56.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:56.542+0000 7f1a4f23d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:56.569 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:56.569 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:56.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:21:58.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~216s 2026-02-25T10:21:58.573 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:21:58.652 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:58.650+0000 7fbb4010b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:58.654 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:21:58.654+0000 7fbb3ee82640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:21:58.680 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:21:58.680 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:21:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:21:58.682 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:00.683 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:00.762 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:00.760+0000 7f358bba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:00.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:00.763+0000 7f358bba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:00.791 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:00.791 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:00.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:01.391 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:01.463 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:01.467 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:02.794 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:02.875 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:02.873+0000 7faebde8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:02.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:02.876+0000 7faebde8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:02.904 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:02.904 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:02.908 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:03.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~222s 2026-02-25T10:22:04.908 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:04.985 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:04.983+0000 7fbcd3774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:04.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:04.987+0000 7fbcd3774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:05.014 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:05.014 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:05.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:06.468 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:06.540 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:06.544 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:07.017 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:07.093 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:07.091+0000 7f43436cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:07.095 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:07.095+0000 7f43436cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:07.122 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:07.123 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:07.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:09.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~227s 2026-02-25T10:22:09.128 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:09.206 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:09.204+0000 7f74efa06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:09.208 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:09.207+0000 7f74efa06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:09.234 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:09.235 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:09.237 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:11.238 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:11.316 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:11.314+0000 7fd9d70b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:11.319 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:11.318+0000 7fd9d70b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:11.347 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:11.348 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:11.350 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:11.544 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:11.617 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:11.620 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:13.350 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:13.429 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:13.427+0000 7fe29b5a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:13.431 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:13.430+0000 7fe29b5a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:13.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:13.459 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:13.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:14.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~232s 2026-02-25T10:22:15.462 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:15.541 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:15.539+0000 7f88eb9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:15.543 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:15.542+0000 7f88eb9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:15.570 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:15.570 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:15.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:16.622 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:16.692 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:16.696 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:17.575 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:17.653 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:17.651+0000 7fa9ceffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:17.655 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:17.654+0000 7fa9ceffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:17.682 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:17.682 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:17.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:19.685 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:19.764 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:19.762+0000 7fc33a15c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:19.766 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:19.766+0000 7fc33a15c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:19.793 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:19.793 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:19.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:19.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~238s 2026-02-25T10:22:21.697 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:21.769 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:21.773 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:21.798 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:21.877 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:21.875+0000 7f68cb1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:21.879 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:21.878+0000 7f68cb1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:21.905 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:21.906 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:21.908 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:23.908 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:23.988 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:23.986+0000 7fa7dad0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:23.990 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:23.990+0000 7fa7dad0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:24.016 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:24.016 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:24.020 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:25.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~243s 2026-02-25T10:22:26.021 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:26.099 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:26.096+0000 7f1a8910b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:26.101 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:26.100+0000 7f1a8910b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:26.127 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:26.127 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:26.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:26.773 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:26.846 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:26.850 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:28.133 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:28.211 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:28.209+0000 7f8736ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:28.214 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:28.213+0000 7f8736ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:28.240 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:28.241 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:28.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:30.244 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:30.323 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:30.321+0000 7fe8695fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:30.325 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:30.324+0000 7fe8695fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:30.353 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:30.353 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:30.355 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:30.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~249s 2026-02-25T10:22:31.851 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:31.923 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:31.926 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:32.357 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:32.436 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:32.434+0000 7f88a7c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:32.438 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:32.438+0000 7f88a7c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:32.465 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:32.465 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:32.469 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:34.470 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:34.549 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:34.547+0000 7f295543d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:34.551 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:34.551+0000 7f295543d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:34.581 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:34.581 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:34.583 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:36.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~254s 2026-02-25T10:22:36.584 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:36.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:36.661+0000 7f511795b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:36.666 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:36.665+0000 7f511795b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:36.693 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:36.693 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:36.695 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:36.927 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:36.998 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:37.001 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:38.696 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:38.775 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:38.773+0000 7f89a2bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:38.777 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:38.777+0000 7f89a2bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:38.804 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:38.805 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:38.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:40.810 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:40.889 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:40.887+0000 7f8cee30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:40.891 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:40.890+0000 7f8cee30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:40.917 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:40.918 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:40.920 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:41.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~259s 2026-02-25T10:22:42.002 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:42.074 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:42.077 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:42.921 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:42.999 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:42.998+0000 7f38d9fe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:43.002 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:43.001+0000 7f38d9fe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:43.029 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:43.029 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:43.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:45.032 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:45.111 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:45.109+0000 7f8a85d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:45.113 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:45.113+0000 7f8a85d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:45.140 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:45.141 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:45.142 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:46.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2026-02-25T10:22:47.078 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:47.143 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:47.149 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:47.153 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:47.222 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:47.220+0000 7f26887ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:47.224 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:47.223+0000 7f26887ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:47.252 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:47.252 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:47.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:49.255 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:49.335 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:49.332+0000 7f01349a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:49.337 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:49.336+0000 7f01349a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:49.365 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:49.365 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:49.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:51.369 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:51.447 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:51.445+0000 7f9f25c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:51.449 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:51.449+0000 7f9f25c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:51.477 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:51.477 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:51.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:52.155 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:52.226 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:52.229 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:52.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2026-02-25T10:22:53.482 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:53.560 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:53.558+0000 7fe0489c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:53.562 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:53.562+0000 7fe047738640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:53.588 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:53.588 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:53.590 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:55.591 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:55.670 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:55.669+0000 7fc8a7acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:55.673 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:55.672+0000 7fc8a7acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:55.700 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:55.700 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:55.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:57.230 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:22:57.301 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:22:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:22:57.304 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:22:57.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~276s 2026-02-25T10:22:57.705 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:22:57.782 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:57.780+0000 7f325eccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:57.784 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:57.784+0000 7f325eccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:57.812 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:57.812 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:57.814 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:22:59.815 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:22:59.892 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:59.890+0000 7f095fe8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:59.894 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:22:59.894+0000 7f095fe8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:22:59.922 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:22:59.922 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:22:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:22:59.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:01.928 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:23:02.005 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:02.003+0000 7ffa57b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:02.008 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:02.007+0000 7ffa57b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:02.035 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:02.035 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:02.037 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:02.305 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:02.375 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:02.378 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:03.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2026-02-25T10:23:04.038 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:23:04.115 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:04.113+0000 7f879465a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:04.118 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:04.117+0000 7f879465a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:04.145 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:04.146 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:04.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:06.149 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:23:06.225 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:06.223+0000 7f4a57f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:06.228 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:06.227+0000 7f4a57f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:06.255 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:06.255 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:06.256 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:07.380 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:07.450 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:07.453 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:08.258 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:23:08.336 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:08.334+0000 7f2ab7e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:08.339 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:08.338+0000 7f2ab7e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:08.365 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:08.365 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:08.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:08.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2026-02-25T10:23:10.370 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:23:10.446 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:10.444+0000 7f9409f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:10.448 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:10.448+0000 7f9409f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:10.475 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:10.475 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:10.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:12.454 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:12.478 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:23:12.524 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:12.528 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:12.557 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:12.555+0000 7f2693c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:12.559 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:12.558+0000 7f2693c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:12.586 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:12.586 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:12.588 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:13.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2026-02-25T10:23:14.589 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:23:14.664 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:14.663+0000 7ff8405e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:14.667 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:14.666+0000 7ff8405e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:14.692 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:14.692 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:14.695 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:16.697 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:23:16.774 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:16.772+0000 7f7888006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:16.776 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:16.776+0000 7f7888006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:16.803 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:16.803 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:16.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:17.529 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:17.600 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:17.603 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:17.604 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:18.806 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:23:18.881 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:18.879+0000 7fb09ca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:18.884 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:18.883+0000 7fb09ca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:18.910 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:18.910 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:18.914 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:19.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2026-02-25T10:23:20.915 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T10:23:20.995 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:20.993+0000 7fe99595c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:20.997 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:20.996+0000 7fe99595c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:21.024 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:21.025 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:21.026 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:22.604 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:22.675 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:22.678 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:23.027 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T10:23:23.104 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:23.102+0000 7f6777d74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:23.107 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:23:23.106+0000 7f6777d74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:23:23.134 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T10:23:23.134 INFO:teuthology.orchestra.run.trial068.stderr:Error ENXIO: problem getting command descriptions from osd.3 2026-02-25T10:23:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T10:23:23.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T10:23:24.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~303s 2026-02-25T10:23:24.718 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2026-02-25T10:23:25.120 INFO:tasks.ceph.osd.0:Sent signal 15 2026-02-25T10:23:25.121 INFO:tasks.ceph.osd.1:Sent signal 15 2026-02-25T10:23:25.121 ERROR:tasks.ceph.osd.3:No such daemon running 2026-02-25T10:23:25.121 INFO:tasks.ceph.mon.a:Sent signal 15 2026-02-25T10:23:25.122 INFO:tasks.ceph.mgr.x:Sent signal 15 2026-02-25T10:23:25.122 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing thrasher OSDThrasher 2026-02-25T10:23:25.122 INFO:tasks.ceph.mgr.x.trial072.stderr:2026-02-25T10:23:25.121+0000 7f45941f8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 39496) UID: 0 2026-02-25T10:23:25.123 INFO:tasks.ceph.mgr.x.trial072.stderr:2026-02-25T10:23:25.121+0000 7f45941f8640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2026-02-25T10:23:25.123 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-25T10:23:25.121+0000 7f40b67fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 40233) UID: 0 2026-02-25T10:23:25.123 INFO:tasks.ceph.mon.a.trial068.stderr:2026-02-25T10:23:25.121+0000 7f40b67fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-25T10:23:27.679 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:27.750 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:27.754 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:32.755 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:32.826 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:32.830 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:37.831 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:37.902 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:37.906 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:42.906 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:42.976 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:42.978 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:47.979 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:48.049 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:48.052 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:53.053 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:53.124 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:53.127 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:23:58.129 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:23:58.199 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:23:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:23:58.223 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:24:03.224 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:24:03.295 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:24:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:24:03.298 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:24:08.299 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:24:08.369 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:24:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:24:08.371 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:24:13.373 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:24:13.443 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:24:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:24:13.447 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:24:18.447 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:24:18.519 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:24:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:24:18.522 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T10:24:23.524 DEBUG:teuthology.orchestra.run.trial072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T10:24:23.594 INFO:teuthology.orchestra.run.trial072.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T10:24:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T10:24:23.602 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-25T10:24:23.602 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1384, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-25T10:24:23.603 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T10:24:23.604 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing remote process ceph-test-rados-ceph 2026-02-25T10:24:23.604 DEBUG:tasks.rados:Stopping ceph-test-rados-ceph due to exception The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T10:24:23.604 INFO:tasks.rados:Stopping instance 0 2026-02-25T10:24:23.616 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T10:24:23.618 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T10:24:23.621 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2026-02-25T10:24:23.639 INFO:tasks.thrashosds:joining thrashosds 2026-02-25T10:24:23.639 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T10:24:23.639 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T10:24:23.639 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T10:24:23.734 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:24:23.731+0000 7f61e3b74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:26:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T10:26:23.690 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2970, in wait_for_all_osds_up while not self.are_all_osds_up(): ^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2960, in are_all_osds_up x = self.get_osd_dump() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2670, in get_osd_dump return self.get_osd_dump_json()['osds'] ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2662, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial068 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2026-02-25T10:26:23.696 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T10:26:23.712 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T10:26:23.713 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T10:26:23.792 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-25T10:26:23.789+0000 7f52d970b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T10:28:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T10:28:23.749 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial068 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T10:28:23.755 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T10:28:23.755 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T10:28:23.755 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T10:28:23.756 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T10:28:23.820 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T10:28:23.820 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T10:28:23.820 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T10:28:23.889 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T10:28:23.889 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T10:28:23.889 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T10:28:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:28:23.890 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial068 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial072 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2026-02-25T10:28:23.892 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T10:28:23.892 ERROR:tasks.ceph.osd.3:tried to stop a non-running daemon 2026-02-25T10:28:23.892 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T10:28:23.893 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T10:28:23.893 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T10:28:23.893 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T10:28:23.893 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T10:28:23.893 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T10:28:23.893 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T10:28:23.893 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T10:28:23.893 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T10:28:23.893 DEBUG:teuthology.orchestra.run.trial068:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v 'but it is still running' | grep -E -v 'objects unfound and apparently lost' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_' | grep -E -v '\(CACHE_POOL_' | grep -E -v '\(SMALLER_PGP_NUM\)' | grep -E -v '\(OBJECT_' | grep -E -v SLOW_OPS | grep -E -v '\(REQUEST_SLOW\)' | grep -E -v '\(TOO_FEW_PGS\)' | grep -E -v 'slow request' | grep -E -v 'timeout on replica' | grep -E -v 'late reservation from' | grep -E -v MON_DOWN | grep -E -v OSDMAP_FLAGS | grep -E -v OSD_DOWN | grep -E -v PG_DEGRADED | grep -E -v PG_AVAILABILITY | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'mons down' | grep -E -v 'mon down' | grep -E -v 'out of quorum' | grep -E -v noscrub | grep -E -v nodeep-scrub | grep -E -v 'Degraded data redundancy' | grep -E -v 'is down' | grep -E -v 'osds down' | grep -E -v 'pg .*? is .*?degraded.*?, acting' | grep -E -v 'pg .*? is stuck' | grep -E -v 'pg degraded' | grep -E -v PG_BACKFILL_FULL | grep -E -v 'Low space hindering backfill .*? backfill_toofull' | grep -E -v OSD_ROOT_DOWN | grep -E -v 'pgs degraded' | grep -E -v 'pgs undersized' | grep -E -v 'is active.*backfill_toofull.*' | head -n 1 2026-02-25T10:28:23.920 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:28:23.920 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T10:28:23.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:28:23.987 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T10:28:24.054 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:28:24.054 DEBUG:teuthology.orchestra.run.trial072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T10:28:24.095 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:28:24.095 DEBUG:teuthology.orchestra.run.trial072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-25T10:28:24.160 INFO:tasks.ceph:Archiving mon data... 2026-02-25T10:28:24.163 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115/data/mon.a.tgz 2026-02-25T10:28:24.163 DEBUG:teuthology.orchestra.run.trial068:> mktemp 2026-02-25T10:28:24.180 INFO:teuthology.orchestra.run.trial068.stdout:/tmp/tmp.dPf5B6uoOQ 2026-02-25T10:28:24.180 DEBUG:teuthology.orchestra.run.trial068:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dPf5B6uoOQ 2026-02-25T10:28:24.278 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0666 /tmp/tmp.dPf5B6uoOQ 2026-02-25T10:28:24.359 DEBUG:teuthology.orchestra.remote:trial068:/tmp/tmp.dPf5B6uoOQ is 293KB 2026-02-25T10:28:24.453 DEBUG:teuthology.orchestra.run.trial068:> rm -fr /tmp/tmp.dPf5B6uoOQ 2026-02-25T10:28:24.469 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T10:28:24.470 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T10:28:24.510 DEBUG:teuthology.orchestra.run.trial072:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T10:28:24.589 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T10:28:24.605 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T10:28:24.605 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T10:28:24.607 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115/remote/trial068/crash 2026-02-25T10:28:24.608 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T10:28:24.654 DEBUG:teuthology.misc:Transferring archived files from trial072:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115/remote/trial072/crash 2026-02-25T10:28:24.656 DEBUG:teuthology.orchestra.run.trial072:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T10:28:24.681 INFO:tasks.ceph:Compressing logs... 2026-02-25T10:28:24.682 DEBUG:teuthology.orchestra.run.trial068:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T10:28:24.700 DEBUG:teuthology.orchestra.run.trial072:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T10:28:24.721 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44211.log 2026-02-25T10:28:24.721 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43150.log 2026-02-25T10:28:24.721 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49299.log 2026-02-25T10:28:24.722 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44211.log.gz 2026-02-25T10:28:24.722 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47391.log 2026-02-25T10:28:24.722 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43150.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43150.log.gz 2026-02-25T10:28:24.722 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49299.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49299.log.gz 2026-02-25T10:28:24.722 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47391.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47391.log.gz 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45218.log 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49246.log 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45218.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45218.log.gz 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49246.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49246.log.gz 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46384.log 2026-02-25T10:28:24.723 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49034.log/var/log/ceph/ceph-osd.0.log: 2026-02-25T10:28:24.724 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40670.log 2026-02-25T10:28:24.724 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46384.log.gz 2026-02-25T10:28:24.724 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46172.log 2026-02-25T10:28:24.724 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49034.log.gz 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45801.log 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40670.log.gz 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44741.log 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46172.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46172.log.gz 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42289.log 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45801.log.gz 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44157.log 2026-02-25T10:28:24.725 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44741.log.gz 2026-02-25T10:28:24.726 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42289.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.47603.log -- replaced with /var/log/ceph/ceph-client.admin.42289.log.gz 2026-02-25T10:28:24.726 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-25T10:28:24.726 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48080.log 2026-02-25T10:28:24.726 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44157.log.gz 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42028.log 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47603.log.gz 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42832.log 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48080.log.gz 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43309.log 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42028.log.gz 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42832.log: -- /var/log/ceph/ceph-client.admin.45059.log 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42832.log.gz 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48133.log 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43309.log.gz 2026-02-25T10:28:24.727 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48875.log 2026-02-25T10:28:24.728 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45059.log.gz 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48133.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48133.log.gz 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42726.log 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49352.log 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48875.log.gz 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42726.log.gz 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47285.log 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.49352.log.gz 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41923.log 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45695.log 2026-02-25T10:28:24.729 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47285.log.gz 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43203.log 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41923.log.gz 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45695.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45695.log.gz 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43786.log 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43203.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43203.log.gz 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46543.log 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43839.log 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43786.log.gz 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43998.log 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46543.log.gz 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43839.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.43839.log.gz --verbose 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr: -- /var/log/ceph/ceph-client.admin.44370.log 2026-02-25T10:28:24.730 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2026-02-25T10:28:24.731 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43998.log.gz 2026-02-25T10:28:24.731 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43521.log 2026-02-25T10:28:24.731 INFO:teuthology.orchestra.run.trial068.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.44370.log.gz 2026-02-25T10:28:24.731 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46861.log 2026-02-25T10:28:24.731 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2026-02-25T10:28:24.731 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43574.log 2026-02-25T10:28:24.731 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43521.log.gz 2026-02-25T10:28:24.732 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44317.log 2026-02-25T10:28:24.732 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46861.log.gz 2026-02-25T10:28:24.732 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41722.log 2026-02-25T10:28:24.732 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43574.log.gz 2026-02-25T10:28:24.732 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43468.log 2026-02-25T10:28:24.732 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44317.log.gz 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48663.log 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41722.log.gz 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40603.log 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43468.log.gz 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41673.log 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48663.log.gz 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44688.log 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41673.log.gz 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47709.log 2026-02-25T10:28:24.733 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44688.log.gz 2026-02-25T10:28:24.734 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2026-02-25T10:28:24.734 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2026-02-25T10:28:24.734 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47709.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39175.log 2026-02-25T10:28:24.734 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47709.log.gz 2026-02-25T10:28:24.734 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42233.log 2026-02-25T10:28:24.734 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2026-02-25T10:28:24.734 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.tmp-client.admin.39175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48716.log 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39175.log.gz 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47020.log 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46278.log 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48716.log.gz 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41925.log 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2026-02-25T10:28:24.735 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47868.log 2026-02-25T10:28:24.736 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46278.log.gz 2026-02-25T10:28:24.736 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49193.log 2026-02-25T10:28:24.736 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41925.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41925.log.gz 2026-02-25T10:28:24.736 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47868.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47868.log.gz 2026-02-25T10:28:24.736 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45324.log 2026-02-25T10:28:24.736 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46914.log 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49193.log.gz 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44104.log 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45324.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45324.log.gz 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46914.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46914.log.gz 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43892.log 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49087.log 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44104.log.gz 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41311.log 2026-02-25T10:28:24.737 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43892.log.gz 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49087.log.gz 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41924.log 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41311.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41311.log.gz 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41526.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42514.log 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43627.log 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41924.log.gz 2026-02-25T10:28:24.738 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48398.log 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43627.log.gz 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48292.log 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.48398.log: /var/log/ceph/ceph-client.admin.48451.log 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48398.log.gz 2026-02-25T10:28:24.739 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48239.log 2026-02-25T10:28:24.740 INFO:teuthology.orchestra.run.trial068.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.48292.log.gz 2026-02-25T10:28:24.740 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46490.log 2026-02-25T10:28:24.740 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48451.log.gz 2026-02-25T10:28:24.740 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47497.log 2026-02-25T10:28:24.740 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48239.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48239.log.gz 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46490.log.gz 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47444.log 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47497.log.gz 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48504.log 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47126.log 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47444.log.gz 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48504.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.48504.log.gz -5 2026-02-25T10:28:24.741 INFO:teuthology.orchestra.run.trial068.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45271.log 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47338.log 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47126.log.gz 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47656.log 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45271.log.gz 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47338.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47338.log.gz 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45430.log 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40740.log 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47656.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47656.log.gz 2026-02-25T10:28:24.742 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45430.log.gz 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40740.log.gz 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41115.log: gzip -5 --verbose 50.0% -- /var/log/ceph/ceph-client.admin.42461.log -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40987.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41428.log 2026-02-25T10:28:24.743 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48822.log 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42461.log.gz 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr:gzip/var/log/ceph/ceph-client.admin.41428.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43256.log 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41428.log.gz 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42938.log 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48822.log.gz 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43256.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43256.log.gz 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49140.log 2026-02-25T10:28:24.744 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42938.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42938.log.gz 2026-02-25T10:28:24.745 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41313.log 2026-02-25T10:28:24.745 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49140.log: gzip -5 --verbose 50.4% -- -- replaced with /var/log/ceph/ceph-client.admin.49140.log.gz 2026-02-25T10:28:24.745 INFO:teuthology.orchestra.run.trial068.stderr: /var/log/ceph/ceph-client.admin.46119.log 2026-02-25T10:28:24.745 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41313.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41313.log.gz 2026-02-25T10:28:24.745 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48981.log 2026-02-25T10:28:24.745 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45483.log 2026-02-25T10:28:24.746 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46119.log.gz 2026-02-25T10:28:24.746 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48981.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48981.log.gz 2026-02-25T10:28:24.746 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46596.log 2026-02-25T10:28:24.746 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47762.log 2026-02-25T10:28:24.746 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45483.log.gz 2026-02-25T10:28:24.746 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44847.log 2026-02-25T10:28:24.746 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46596.log.gz 2026-02-25T10:28:24.747 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47762.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47762.log.gz 2026-02-25T10:28:24.747 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49406.log 2026-02-25T10:28:24.747 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44847.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T10:28:24.747 INFO:teuthology.orchestra.run.trial068.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.44847.log.gz 2026-02-25T10:28:24.747 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T10:28:24.748 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.49406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45642.log 2026-02-25T10:28:24.748 INFO:teuthology.orchestra.run.trial068.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.49406.log.gz 2026-02-25T10:28:24.748 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44582.log 2026-02-25T10:28:24.748 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44051.log 2026-02-25T10:28:24.748 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45642.log.gz 2026-02-25T10:28:24.748 INFO:teuthology.orchestra.run.trial068.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/ceph-client.admin.39748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39691.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39643.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/ceph-client.admin.39691.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39691.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/ceph-client.admin.39643.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39643.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40369.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44582.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45907.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44051.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40369.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40369.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45907.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45907.log.gz 2026-02-25T10:28:24.750 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47550.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46967.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47550.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47550.log.gz 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48769.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43733.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46967.log.gz 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40889.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48769.log.gz 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42673.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43733.log.gz 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48186.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45377.log 49.6% 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42673.log.gz 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45589.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48186.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48186.log.gz 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45377.log: /var/log/ceph/ceph-client.admin.48928.log 2026-02-25T10:28:24.751 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45377.log.gz 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40938.log 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45589.log.gz 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41309.log 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48928.log.gz 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40938.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40938.log.gz 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44264.log 2026-02-25T10:28:24.752 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41309.log.gz 2026-02-25T10:28:24.753 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47921.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2026-02-25T10:28:24.753 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48557.log 2026-02-25T10:28:24.753 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44264.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44264.log.gz -5 2026-02-25T10:28:24.753 INFO:teuthology.orchestra.run.trial068.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43680.log 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45006.log 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.48557.log.gz 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47232.log 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43680.log.gz 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45006.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45006.log.gz 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45536.log 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40767.log 2026-02-25T10:28:24.754 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47232.log.gz 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45536.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45536.log.gz 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45112.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41477.log 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46437.log 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2026-02-25T10:28:24.755 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48027.log 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41477.log.gz 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42885.log 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46437.log.gz 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48027.log.gz 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42175.log 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42885.log.gz 2026-02-25T10:28:24.756 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41117.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2026-02-25T10:28:24.757 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47974.log 2026-02-25T10:28:24.757 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2026-02-25T10:28:24.757 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42175.log.gz 2026-02-25T10:28:24.757 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47974.log: gzip -5 50.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.47974.log.gz -- 2026-02-25T10:28:24.757 INFO:teuthology.orchestra.run.trial068.stderr: /var/log/ceph/ceph-client.admin.42620.log 2026-02-25T10:28:24.757 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40236.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44635.log 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42620.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42620.log.gz 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43097.log 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48345.log 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44635.log.gz 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43097.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43097.log.gz 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44900.log 2026-02-25T10:28:24.758 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44529.log 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48345.log.gz 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44900.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44900.log.gz 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41798.log 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46225.log 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44529.log.gz 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41798.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41798.log.gz 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42779.log 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42131.log 2026-02-25T10:28:24.759 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46225.log.gz 2026-02-25T10:28:24.760 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42779.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42779.log.gz 2026-02-25T10:28:24.760 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43415.log 2026-02-25T10:28:24.760 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.0.42131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40577.log 2026-02-25T10:28:24.760 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.0.42131.log.gz 2026-02-25T10:28:24.760 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43415.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43415.log.gz 2026-02-25T10:28:24.760 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47179.log 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40577.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40577.log.gz 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47179.log: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.47179.log.gz 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44476.log 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44476.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44476.log.gz 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42077.log 2026-02-25T10:28:24.761 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45960.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42567.log 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42077.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42077.log.gz 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48610.log 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr: 91.4% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42567.log.gz 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.40322.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42408.log 2026-02-25T10:28:24.762 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.48610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45854.log 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48610.log.gz 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42408.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42408.log.gz 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44953.log 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45854.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45854.log.gz 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46649.log 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44953.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44953.log.gz 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46331.log 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42991.log 48.0% 2026-02-25T10:28:24.763 INFO:teuthology.orchestra.run.trial068.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46649.log.gz 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46331.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46331.log.gz 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47815.log 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47073.log 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42991.log.gz 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47815.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47815.log.gz 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46702.log 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.47073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46013.log 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47073.log.gz 2026-02-25T10:28:24.764 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46702.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46702.log.gz 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43362.log 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44423.log 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.46013.log.gz 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43362.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43362.log.gz 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44794.log 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44423.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44423.log.gz 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2026-02-25T10:28:24.765 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.44794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43044.log 2026-02-25T10:28:24.766 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44794.log.gz 2026-02-25T10:28:24.766 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.45165.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2026-02-25T10:28:24.766 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2026-02-25T10:28:24.766 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.43044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46808.log 2026-02-25T10:28:24.766 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43044.log.gz 2026-02-25T10:28:24.766 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41575.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2026-02-25T10:28:24.766 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42355.log 2026-02-25T10:28:24.767 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41116.log 2026-02-25T10:28:24.767 INFO:teuthology.orchestra.run.trial068.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46808.log.gz 2026-02-25T10:28:24.767 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.42355.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42355.log.gz 2026-02-25T10:28:24.767 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46066.log 2026-02-25T10:28:24.767 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.41116.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41116.log.gz 2026-02-25T10:28:24.767 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-client.admin.46066.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46066.log.gz 2026-02-25T10:28:24.778 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/ceph-mgr.x.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T10:28:24.825 INFO:teuthology.orchestra.run.trial068.stderr: 89.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T10:28:25.163 INFO:teuthology.orchestra.run.trial072.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-25T10:28:25.169 INFO:teuthology.orchestra.run.trial068.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T10:28:25.221 INFO:teuthology.orchestra.run.trial068.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T10:28:25.222 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-25T10:28:25.222 INFO:teuthology.orchestra.run.trial068.stderr:real 0m0.510s 2026-02-25T10:28:25.222 INFO:teuthology.orchestra.run.trial068.stderr:user 0m1.031s 2026-02-25T10:28:25.222 INFO:teuthology.orchestra.run.trial068.stderr:sys 0m0.118s 2026-02-25T10:28:25.279 INFO:teuthology.orchestra.run.trial072.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T10:28:25.281 INFO:teuthology.orchestra.run.trial072.stderr: 2026-02-25T10:28:25.281 INFO:teuthology.orchestra.run.trial072.stderr:real 0m0.541s 2026-02-25T10:28:25.281 INFO:teuthology.orchestra.run.trial072.stderr:user 0m0.925s 2026-02-25T10:28:25.281 INFO:teuthology.orchestra.run.trial072.stderr:sys 0m0.069s 2026-02-25T10:28:25.281 INFO:tasks.ceph:Archiving logs... 2026-02-25T10:28:25.282 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115/remote/trial068/log 2026-02-25T10:28:25.283 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T10:28:25.596 DEBUG:teuthology.misc:Transferring archived files from trial072:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115/remote/trial072/log 2026-02-25T10:28:25.597 DEBUG:teuthology.orchestra.run.trial072:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T10:28:25.770 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial068 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T10:28:25.772 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T10:28:25.783 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T10:28:25.904 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T10:28:25.916 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T10:28:25.917 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T10:28:25.917 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T10:28:25.919 DEBUG:teuthology.orchestra.run.trial072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T10:28:25.944 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T10:28:25.956 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T10:28:25.956 DEBUG:teuthology.orchestra.run.trial068:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T10:28:25.984 DEBUG:teuthology.orchestra.run.trial072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T10:28:25.998 INFO:teuthology.orchestra.run.trial068.stderr:bash: line 1: ntpq: command not found 2026-02-25T10:28:25.999 INFO:teuthology.orchestra.run.trial072.stderr:bash: line 1: ntpq: command not found 2026-02-25T10:28:26.024 INFO:teuthology.orchestra.run.trial068.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T10:28:26.024 INFO:teuthology.orchestra.run.trial068.stdout:=============================================================================== 2026-02-25T10:28:26.024 INFO:teuthology.orchestra.run.trial068.stdout:^+ rn-02.koehn.com 2 6 377 250 -109us[ +39us] +/- 11ms 2026-02-25T10:28:26.024 INFO:teuthology.orchestra.run.trial068.stdout:^- 149.28.61.105.vultruserc> 3 6 377 59 +938us[ +969us] +/- 32ms 2026-02-25T10:28:26.024 INFO:teuthology.orchestra.run.trial068.stdout:^* time.cloudflare.com 3 6 377 58 +171us[ +202us] +/- 6923us 2026-02-25T10:28:26.024 INFO:teuthology.orchestra.run.trial068.stdout:^- frumentum-zeus.infra.fru> 2 6 377 57 -1407us[-1407us] +/- 24ms 2026-02-25T10:28:26.026 INFO:teuthology.orchestra.run.trial072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T10:28:26.026 INFO:teuthology.orchestra.run.trial072.stdout:=============================================================================== 2026-02-25T10:28:26.026 INFO:teuthology.orchestra.run.trial072.stdout:^+ rn-02.koehn.com 2 6 377 59 +53us[-5567ns] +/- 11ms 2026-02-25T10:28:26.026 INFO:teuthology.orchestra.run.trial072.stdout:^- 149.28.61.105.vultruserc> 3 6 377 59 +1091us[+1033us] +/- 32ms 2026-02-25T10:28:26.026 INFO:teuthology.orchestra.run.trial072.stdout:^* time.cloudflare.com 3 6 377 57 +9417ns[ -49us] +/- 6981us 2026-02-25T10:28:26.026 INFO:teuthology.orchestra.run.trial072.stdout:^- frumentum-zeus.infra.fru> 2 6 377 58 -1390us[-1449us] +/- 24ms 2026-02-25T10:28:26.027 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T10:28:26.039 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T10:28:26.040 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T10:28:26.055 DEBUG:teuthology.orchestra.run.trial068:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T10:28:26.123 DEBUG:teuthology.orchestra.run.trial072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T10:28:26.179 DEBUG:teuthology.orchestra.run.trial068:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T10:28:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:28:26.207 DEBUG:teuthology.orchestra.run.trial072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T10:28:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:28:26.233 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T10:28:26.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T10:28:26.262 INFO:teuthology.task.internal:Duration was 1011.229949 seconds 2026-02-25T10:28:26.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T10:28:26.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T10:28:26.277 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T10:28:26.280 DEBUG:teuthology.orchestra.run.trial072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T10:28:26.315 INFO:teuthology.orchestra.run.trial068.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T10:28:26.316 INFO:teuthology.orchestra.run.trial072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T10:28:26.743 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T10:28:26.743 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial068.front.sepia.ceph.com 2026-02-25T10:28:26.743 DEBUG:teuthology.orchestra.run.trial068:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T10:28:26.764 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial072.front.sepia.ceph.com 2026-02-25T10:28:26.765 DEBUG:teuthology.orchestra.run.trial072:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T10:28:26.801 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T10:28:26.802 DEBUG:teuthology.orchestra.run.trial068:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T10:28:26.807 DEBUG:teuthology.orchestra.run.trial072:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T10:28:26.905 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T10:28:26.905 DEBUG:teuthology.orchestra.run.trial068:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T10:28:26.907 DEBUG:teuthology.orchestra.run.trial072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T10:28:26.929 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T10:28:26.929 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T10:28:26.929 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T10:28:26.929 INFO:teuthology.orchestra.run.trial068.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T10:28:26.929 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T10:28:26.931 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T10:28:26.932 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T10:28:26.932 INFO:teuthology.orchestra.run.trial072.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T10:28:26.932 INFO:teuthology.orchestra.run.trial072.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T10:28:26.933 INFO:teuthology.orchestra.run.trial072.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T10:28:26.939 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T10:28:26.943 INFO:teuthology.orchestra.run.trial072.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.6% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T10:28:26.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T10:28:26.963 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T10:28:26.963 DEBUG:teuthology.orchestra.run.trial068:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T10:28:27.005 DEBUG:teuthology.orchestra.run.trial072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T10:28:27.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T10:28:27.048 DEBUG:teuthology.orchestra.run.trial068:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T10:28:27.050 DEBUG:teuthology.orchestra.run.trial072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T10:28:27.071 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern = core 2026-02-25T10:28:27.093 INFO:teuthology.orchestra.run.trial072.stdout:kernel.core_pattern = core 2026-02-25T10:28:27.107 DEBUG:teuthology.orchestra.run.trial068:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T10:28:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:28:27.143 DEBUG:teuthology.orchestra.run.trial072:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T10:28:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T10:28:27.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T10:28:27.183 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T10:28:27.185 DEBUG:teuthology.misc:Transferring archived files from trial068:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115/remote/trial068 2026-02-25T10:28:27.185 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T10:28:27.216 DEBUG:teuthology.misc:Transferring archived files from trial072:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69115/remote/trial072 2026-02-25T10:28:27.216 DEBUG:teuthology.orchestra.run.trial072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T10:28:27.251 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T10:28:27.251 DEBUG:teuthology.orchestra.run.trial068:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T10:28:27.256 DEBUG:teuthology.orchestra.run.trial072:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T10:28:27.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T10:28:27.312 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T10:28:27.312 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T10:28:27.326 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T10:28:27.326 DEBUG:teuthology.orchestra.run.trial068:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T10:28:27.327 DEBUG:teuthology.orchestra.run.trial072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T10:28:27.341 INFO:teuthology.orchestra.run.trial068.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 10:28 /home/ubuntu/cephtest 2026-02-25T10:28:27.355 INFO:teuthology.orchestra.run.trial072.stdout: 917525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 10:28 /home/ubuntu/cephtest 2026-02-25T10:28:27.356 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T10:28:27.370 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T10:28:27.389 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T10:28:27.394 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} thrashers/{default} thrashosds-health workloads/write_fadvise_dontneed} duration: 1011.2299494743347 failure_reason: The WatchDog barked due to Daemon ceph.osd.2 has failed flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com status: fail success: false 2026-02-25T10:28:27.394 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T10:28:27.500 INFO:teuthology.run:FAIL